%% LRaven1/matlab/paramsearch/cvR2opt-bruteforce-35x28-vega-20110712.txt %% [AAP note 2011-07-12] The new initialization method did not live up to our expectations %% and was abandoned. Thus, the following is merely a replication of the cvR2=.41 run %% with the old (all-zero) initialization. Replication of cvR2opt-bruteforce-35x28-vega-20110404.txt. assessing_cv_method executed on 12-Jul-2011 15:48:26. cd /Users/apetrov/a/r/w/work/RelExper/LRaven1/matlab/paramsearch load /Users/apetrov/a/r/w/work/RelExper/LRaven1/matlab/data/H.mat load /Users/apetrov/a/r/w/work/RelExper/LRaven1/matlab/data/transitions/collapsed_clip_205.mat D = sequences: {35x28 cell} descriptor: 'Collapsed sequences loaded from data/transitions/collapsed_clip_205.mat' session1_score: [35x1 double] session2_score: [35x1 double] session12_score: [35x1 double] target: [35x1 double] Value Count Percent Cum_cnt Cum_pct ------------------------------------------- 12 1 2.86 1 2.86 13 1 2.86 2 5.71 15 1 2.86 3 8.57 16 1 2.86 4 11.43 18 1 2.86 5 14.29 19 2 5.71 7 20.00 20 3 8.57 10 28.57 21 4 11.43 14 40.00 22 3 8.57 17 48.57 23 2 5.71 19 54.29 24 7 20.00 26 74.29 25 3 8.57 29 82.86 26 5 14.29 34 97.14 27 1 2.86 35 100.00 ------------------------------------------- ans = Display: 'off' MaxFunEvals: [] MaxIter: 30 TolFun: 1.0000e-06 TolX: [] FunValCheck: 'off' OutputFcn: [] PlotFcns: [] ActiveConstrTol: [] Algorithm: 'trust-region-reflective' AlwaysHonorConstraints: 'bounds' BranchStrategy: [] DerivativeCheck: 'off' Diagnostics: 'off' DiffMaxChange: 0.1000 DiffMinChange: 1.0000e-08 FinDiffType: 'forward' GoalsExactAchieve: [] GradConstr: 'off' GradObj: 'off' HessFcn: [] Hessian: [] HessMult: [] HessPattern: 'sparse(ones(numberofvariables))' HessUpdate: [] InitialHessType: [] InitialHessMatrix: [] InitBarrierParam: 0.1000 InitTrustRegionRadius: 'sqrt(numberofvariables)' Jacobian: [] JacobMult: [] JacobPattern: [] LargeScale: 'on' LevenbergMarquardt: [] LineSearchType: [] MaxNodes: [] MaxPCGIter: 'max(1,floor(numberofvariables/2))' MaxProjCGIter: '2*(numberofvariables-numberofequalities)' MaxRLPIter: [] MaxSQPIter: '10*max(numberofvariables,numberofinequalities+numberofbounds)' MaxTime: [] MeritFunction: [] MinAbsMax: [] NodeDisplayInterval: [] NodeSearchStrategy: [] NonlEqnAlgorithm: [] NoStopIfFlatInfeas: 'off' ObjectiveLimit: -1.0000e+20 PhaseOneTotalScaling: 'off' Preconditioner: [] PrecondBandWidth: 0 RelLineSrchBnd: [] RelLineSrchBndDuration: 1 ScaleProblem: 'obj-and-constr' Simplex: [] SubproblemAlgorithm: 'ldl-factorization' TolCon: 1.0000e-06 TolConSQP: 1.0000e-06 TolGradCon: 1.0000e-06 TolPCG: 0.1000 TolProjCG: 0.0100 TolProjCGAbs: 1.0000e-10 TolRLPFun: [] TolXInteger: [] TypicalX: 'ones(numberofvariables,1)' UseParallel: 'never' search_params = learn_function: @successrep_learn N_AOIs: 10 N_comp: 2 max_N_comp: 12 sbj_idx: [1x35 double] trial_idx: [1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28] N_left_out: 0 Borda_diagnosep: 0 N_random_seeds: 200 N_searches: 5 suggested_seeds: [0.2500 0 0.2300] N_left_outermost: 1 N_carryover_seeds: 0 verbosep: 0 parallelp: 1 options: [1x1 struct] log_streams: 1 log_title: 'Reproduce the run with cvR2=.41, 2011-07-12' gamma_bounds: [0.1500 0.3500] lambda_gamma_bounds: [0 0] lrate_bounds: [0.1500 0.3500] >> RR = LRaven1_R2_optim(D,search_params) *** LRaven1_R2_optim executed on vega, 20110712T154847 *** Log title: Reproduce the run with cvR2=.41, 2011-07-12 Data descriptor: Collapsed sequences loaded from data/transitions/collapsed_clip_205.mat There are sequences for 35 subjects x 28 trials. Some may be NaNs. Subjects in the outermost loop: [1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35] Trial index: [1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28] Suggested seed(s): [0.25 0 0.23] 200 random searches per set. 5 FMINSEARCHCON per set. 0 seeds carried across sets. Performing 35 cross-validation iterations... 1: Lv 35: gm=0.272 lg=0.000 lr=0.227, R2=0.538, cvR2=0.538, cvRMSE=2.546. PC [1 7]: R2g=[0.316 0.222], s=[1 0.94]. R2=0.538, Y=21, Yhat=20.6 2: Lv 34: gm=0.236 lg=0.000 lr=0.234, R2=0.552, cvR2=0.552, cvRMSE=2.509. PC [1 7]: R2g=[0.319 0.233], s=[1 0.94]. R2=0.552, Y=22, Yhat=23.9 3: Lv 33: gm=0.251 lg=0.000 lr=0.227, R2=0.515, cvR2=0.515, cvRMSE=2.598. PC [1 7]: R2g=[0.303 0.212], s=[1 0.93]. R2=0.515, Y=24, Yhat=24.2 4: Lv 32: gm=0.248 lg=0.000 lr=0.224, R2=0.552, cvR2=0.552, cvRMSE=2.498. PC [1 7]: R2g=[0.301 0.25], s=[1 0.94]. R2=0.552, Y=24, Yhat=23.6 5: Lv 31: gm=0.216 lg=0.000 lr=0.219, R2=0.563, cvR2=0.563, cvRMSE=2.474. PC [1 7]: R2g=[0.309 0.254], s=[1 0.94]. R2=0.563, Y=23, Yhat=20.7 6: Lv 30: gm=0.215 lg=0.000 lr=0.219, R2=0.573, cvR2=0.573, cvRMSE=2.405. PC [1 7]: R2g=[0.299 0.275], s=[1 0.94]. R2=0.573, Y=26, Yhat=21.8 7: Lv 29: gm=0.297 lg=0.000 lr=0.225, R2=0.584, cvR2=0.584, cvRMSE=2.415. PC [2 7]: R2g=[0.319 0.265], s=[1 0.94]. R2=0.584, Y=21, Yhat=17.7 8: Lv 28: gm=0.252 lg=0.000 lr=0.234, R2=0.557, cvR2=0.557, cvRMSE=2.484. PC [1 7]: R2g=[0.305 0.253], s=[1 0.94]. R2=0.557, Y=20, Yhat=20.6 9: Lv 27: gm=0.230 lg=0.000 lr=0.229, R2=0.575, cvR2=0.575, cvRMSE=2.443. PC [1 7]: R2g=[0.316 0.258], s=[1 0.94]. R2=0.575, Y=21, Yhat=23.4 10: Lv 26: gm=0.298 lg=0.000 lr=0.247, R2=0.575, cvR2=0.575, cvRMSE=2.400. PC [1 7]: R2g=[0.329 0.246], s=[1 0.94]. R2=0.575, Y=26, Yhat=23.6 11: Lv 25: gm=0.209 lg=0.000 lr=0.222, R2=0.578, cvR2=0.578, cvRMSE=2.413. PC [1 7]: R2g=[0.343 0.234], s=[1 0.94]. R2=0.578, Y=19, Yhat=22.9 12: Lv 24: gm=0.256 lg=0.000 lr=0.233, R2=0.565, cvR2=0.565, cvRMSE=2.462. PC [1 7]: R2g=[0.313 0.252], s=[1 0.94]. R2=0.565, Y=24, Yhat=22.5 13: Lv 23: gm=0.336 lg=0.000 lr=0.257, R2=0.557, cvR2=0.557, cvRMSE=2.484. PC [1 7]: R2g=[0.298 0.259], s=[1 0.94]. R2=0.557, Y=24, Yhat=25.6 14: Lv 22: gm=0.255 lg=0.000 lr=0.270, R2=0.492, cvR2=0.492, cvRMSE=2.371. PC [7 1]: R2g=[0.258 0.234], s=[1 0.94]. R2=0.492, Y=12, Yhat=17.3 15: Lv 21: gm=0.252 lg=0.000 lr=0.220, R2=0.562, cvR2=0.562, cvRMSE=2.469. PC [2 7]: R2g=[0.305 0.257], s=[1 0.94]. R2=0.562, Y=24, Yhat=24.9 16: Lv 20: gm=0.291 lg=0.000 lr=0.245, R2=0.531, cvR2=0.531, cvRMSE=2.543. PC [1 7]: R2g=[0.295 0.236], s=[1 0.94]. R2=0.531, Y=25, Yhat=22.7 17: Lv 19: gm=0.176 lg=0.000 lr=0.205, R2=0.525, cvR2=0.525, cvRMSE=2.537. PC [1 7]: R2g=[0.304 0.221], s=[1 0.94]. R2=0.525, Y=26, Yhat=24.6 18: Lv 18: gm=0.236 lg=0.000 lr=0.224, R2=0.561, cvR2=0.561, cvRMSE=2.439. PC [1 7]: R2g=[0.31 0.251], s=[1 0.94]. R2=0.561, Y=26, Yhat=23.9 19: Lv 17: gm=0.251 lg=0.000 lr=0.258, R2=0.546, cvR2=0.546, cvRMSE=2.481. PC [2 7]: R2g=[0.283 0.263], s=[1 0.94]. R2=0.546, Y=26, Yhat=24.7 20: Lv 16: gm=0.260 lg=0.000 lr=0.233, R2=0.545, cvR2=0.545, cvRMSE=2.504. PC [1 7]: R2g=[0.321 0.224], s=[1 0.94]. R2=0.545, Y=25, Yhat=22.6 21: Lv 15: gm=0.213 lg=0.000 lr=0.212, R2=0.541, cvR2=0.541, cvRMSE=2.515. PC [1 7]: R2g=[0.305 0.236], s=[1 0.94]. R2=0.541, Y=25, Yhat=24.5 22: Lv 14: gm=0.238 lg=0.000 lr=0.269, R2=0.605, cvR2=0.605, cvRMSE=2.317. PC [1 7]: R2g=[0.398 0.207], s=[1 0.93]. R2=0.605, Y=18, Yhat=24.7 23: Lv 13: gm=0.340 lg=0.000 lr=0.266, R2=0.576, cvR2=0.576, cvRMSE=2.438. PC [1 7]: R2g=[0.308 0.269], s=[1 0.94]. R2=0.576, Y=21, Yhat=20 24: Lv 12: gm=0.291 lg=0.000 lr=0.243, R2=0.519, cvR2=0.519, cvRMSE=2.500. PC [1 7]: R2g=[0.362 0.157], s=[1 0.93]. R2=0.519, Y=16, Yhat=21.1 25: Lv 11: gm=0.317 lg=0.000 lr=0.269, R2=0.480, cvR2=0.480, cvRMSE=2.691. PC [1 7]: R2g=[0.314 0.166], s=[1 0.93]. R2=0.480, Y=24, Yhat=23.6 26: Lv 10: gm=0.309 lg=0.000 lr=0.255, R2=0.481, cvR2=0.481, cvRMSE=2.688. PC [1 7]: R2g=[0.3 0.181], s=[1 0.93]. R2=0.481, Y=24, Yhat=25.1 27: Lv 9: gm=0.350 lg=0.000 lr=0.241, R2=0.569, cvR2=0.569, cvRMSE=2.330. PC [7 2]: R2g=[0.316 0.253], s=[1 0.94]. R2=0.569, Y=15, Yhat=19.8 28: Lv 8: gm=0.258 lg=0.000 lr=0.235, R2=0.542, cvR2=0.542, cvRMSE=2.536. PC [1 7]: R2g=[0.308 0.234], s=[1 0.94]. R2=0.542, Y=22, Yhat=21.1 29: Lv 7: gm=0.250 lg=0.000 lr=0.224, R2=0.564, cvR2=0.564, cvRMSE=2.407. PC [1 7]: R2g=[0.336 0.228], s=[1 0.94]. R2=0.564, Y=27, Yhat=22.3 30: Lv 6: gm=0.285 lg=0.000 lr=0.247, R2=0.559, cvR2=0.559, cvRMSE=2.481. PC [1 7]: R2g=[0.306 0.253], s=[1 0.94]. R2=0.559, Y=20, Yhat=20.1 31: Lv 5: gm=0.288 lg=0.000 lr=0.242, R2=0.554, cvR2=0.554, cvRMSE=2.482. PC [1 7]: R2g=[0.3 0.254], s=[1 0.94]. R2=0.554, Y=19, Yhat=20.6 32: Lv 4: gm=0.230 lg=0.000 lr=0.225, R2=0.555, cvR2=0.555, cvRMSE=2.499. PC [1 7]: R2g=[0.315 0.239], s=[1 0.94]. R2=0.555, Y=23, Yhat=21.8 33: Lv 3: gm=0.178 lg=0.000 lr=0.203, R2=0.561, cvR2=0.561, cvRMSE=2.475. PC [1 7]: R2g=[0.304 0.257], s=[1 0.94]. R2=0.561, Y=20, Yhat=18.8 34: Lv 2: gm=0.316 lg=0.000 lr=0.252, R2=0.552, cvR2=0.552, cvRMSE=2.508. PC [2 7]: R2g=[0.334 0.218], s=[1 0.93]. R2=0.552, Y=22, Yhat=20.1 35: Lv 1: gm=0.154 lg=0.000 lr=0.210, R2=0.480, cvR2=0.480, cvRMSE=2.459. PC [1 7]: R2g=[0.272 0.208], s=[1 0.93]. R2=0.480, Y=13, Yhat=19.1 Done! Global cvRMSE=2.846, cvR2=0.412. Benchmark RMSE=3.695, cvRMSE_norm=0.770. Time is 20110712T190414 RR = log_title: 'Reproduce the run with cvR2=.41, 2011-07-12' host: 'vega' timestamp: {'20110712T154847' '20110712T190414'} data_descriptor: 'Collapsed sequences loaded from data/transitions/collapsed_clip_205.mat' search_params: [1x1 struct] train_sets: [35x34 double] test_sets: [35x1 double] holdout_subjects: [35x1 double] seeds: {35x1 cell} parsearch_res: [35x1 struct] opt_SR_params: [35x3 double] R2_train: [35x1 double] R2_test: [35x1 double] RMSE_train: [35x1 double] RMSE_test: [35x1 double] goalfun_res: [35x1 struct] beta: [35x3 double] R2_train1: [35x1 double] weights1: [35x100 double] weights: [35x100 double] const_term: [35x1 double] predicted_targets: [35x1 double] observed_targets: [35x1 double] global_cvR2: 0.4119 benchmark_RMSE: 3.6951 global_cvRMSE: 2.8463 global_cvRMSE_norm: 0.7703 ########################################################################################## ########################################################################################## ########################################################################################## ############ This is a copy of the script that generated the transcript above ######### ########################################################################################## ########################################################################################## ########################################################################################## % assessing_cv_method-- Generate search_parameters for LRaven1_R2_optim % cross validation for either cumulative score or % difference score. % % Cumulative score cross validation and difference score cross validation % are set by the search_parameter_diff. True = difference scores and false % = cumulative scores. % % See also LRaven1_R2_optim, LRaven1_gridcomp_optim % (c) Laboratory for Cognitive Modeling and Computational Cognitive % Neuroscience at the Ohio State University, http://cogmod.osu.edu % % 1.0.1 2011-07-12 AAP: seq_filename automatically recorded in D.descriptor % Explicit search_params.learn_function = @successrep_learn % 1.0.0 2011-03-30 TRH: Used as helper to quickly create cv search params %% %%%%%%%%%%%%%%%%%%%%%% RUN CUMULATIVE SCORE CV %%%%%%%%%%%%%%%%%%%%%%%%% % 010: Get started cd(fullfile(LRaven1_pathstr,'paramsearch')) ; fprintf('\n\n assessing_cv_method executed on %s.\n\n',datestr(now)) ; fprintf('cd %s\n',pwd) ; clear all ; close all ; %% 020: Load the structure H with the Raven scores % Load H: <1x35 struct> produced by LRaven1_hitcount filename = fullfile(LRaven1_pathstr,'data','H.mat') ; fprintf('load %s \n\n',filename) ; load(filename) ; clear filename %% 025: Load sequence data % Load sequence data: <35x28 cell array> of transition sequence vectors seq_filename = fullfile('data','transitions','collapsed_clip_205.mat') ; filename = fullfile(LRaven1_pathstr,seq_filename) ; fprintf('load %s \n\n',filename) ; load(filename) ; clearvars filename %% 030: Create D structure % Pack sequences into D D.sequences = recoded_sequences ; % Provide descriptor or sequences D.descriptor = ['Collapsed sequences loaded from ' seq_filename] ; % Copy various possible score targets from H to D % These are not going to be used by LRaven1_R2_optim.m. Only D.target is % used. The rest are just stored along for the ride, and for post-hoc % analyses. D.session1_score = [H.session1_score]' ; D.session2_score = [H.session2_score]' ; D.session12_score = [H.session1_score]' + [H.session2_score]' ; % Single out which schores are TARGET during this particular run. D.target = D.session12_score ; % Print in the published transcript D %#ok xtab1(D.target) % Size up the data -- SHOULD ALWAYS BE 35 x 28 [N_sbj,N_trials] = size(D.sequences) ; %% 040: Create search_params structure %- Learning function and its associated fields: search_params.learn_function = @successrep_learn ; N_AOIs = 10 ; search_params.N_AOIs = N_AOIs ; % Number of unique interest areas %- Principal-component fields N_comp = 2 ; search_params.N_comp = N_comp ; % Number of components % all sbjs search_params.max_N_comp = 12 ; % Maximum number of possible components % 12 sbj or less %search_params.max_N_comp = 6 ; % Maximum number of possible components %- Subject and trial fields %search_params.sbj_idx = (1:2:N_sbj) ; % quick-and-dirty debugging search_params.sbj_idx = 1:N_sbj ; % Sbjs for analysis search_params.trial_idx = 1:28 ; % Trials for analysis %- Search and Cross Validation fields search_params.N_left_out = 0 ; % Number of sbj left out for cross validation search_params.Borda_diagnosep = false ; % passed to select_components_borda.m search_params.N_random_seeds = 200 ; % Number of random param searches %search_params.N_random_seeds = 10 ; % quick-and-dirty debugging search_params.N_searches = 5 ; % Number of top seeds used for simplex run search_params.suggested_seeds = [0.2500 0 0.2300] ; % [gamma lambda_gamma lrate] search_params.N_left_outermost = 1 ; % Cross valid. #sbj outer loop search_params.N_carryover_seeds = 0 ; % Number of seeds used in next iter. %- Options and other fields search_params.verbosep = false ; % whether to display progress messages search_params.parallelp = true ; % whether to use parallel processing options = optimset('fmincon') ; % Set default options %- Turn off display for parallel processing and set iterations to 30 search_params.options = optimset(options,'Display','off','MaxIter',30) ; %- File names %fname_stem = 'vega-20110712' ; %fname_stem = 'delete-me' ; % use for debug search_params.log_streams = [1] ; %#ok % only the console unless appended below search_params.log_title = 'Reproduce the run with cvR2=.41, 2011-07-12' ; %- Set bounds on parameters search_params.gamma_bounds = [.15 .35] ; search_params.lambda_gamma_bounds = [0 0] ; search_params.lrate_bounds = [.15 .35] ; %- Display search_params in the published transcript search_params.options search_params %#ok %% 050: Call R2 optim RR = LRaven1_R2_optim(D,search_params) %#ok