前版，实线卷积核使用1*1和3*3，三次虚线残差,dropout=0.3，得分79.35！
试了一上午，去掉标签平滑，勉强得了一个80.01分！均值和方差不好看！
前头版本，实线卷积核使用3*3和3*3，效果不好，午饭时，竟然可以了！
而且冲击80分成功！dropout=0.2！很稳！三次虚线残差,（没有标签平滑）均值方差也很稳！
也就是说实线卷积核使用3*3和3*3也成功了！
轮次：12
learn rate：0.001
rb均值: 0.8941338658,rb方差:5.362059116364
rb均值: 2.7571389675,rb方差:8.011943817139
rb均值: -1.3482213020,rb方差:6.152009963989
rb均值: 0.7353081703,rb方差:5.135601043701
rb均值: -0.5397194028,rb方差:6.214303016663
rb均值: -4.2707591057,rb方差:5.045800209045
rb均值: 0.0182875115,rb方差:5.157238960266
时间: 33650.457031 ms
train Classification result: 82.35% ok (used 49984 images)
时间: 2513.154053 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：13
learn rate：0.001
rb均值: -1.7647972107,rb方差:5.376667022705
rb均值: 1.1370108128,rb方差:7.170983791351
rb均值: 2.9774935246,rb方差:10.114371299744
rb均值: -1.5830640793,rb方差:8.383977890015
rb均值: 1.0266435146,rb方差:6.511214256287
rb均值: 5.1217021942,rb方差:5.058464050293
rb均值: -2.0805156231,rb方差:5.700444698334
rb均值: -0.4777100384,rb方差:7.957851409912
rb均值: -0.6557652950,rb方差:5.065405845642
rb均值: -4.1403393745,rb方差:5.364099025726
rb均值: 0.0164263882,rb方差:6.080315113068
时间: 33687.726562 ms
train Classification result: 83.89% ok (used 49984 images)
时间: 2549.167969 ms
Test Classification result: 10.02% ok (used 9984 images)
轮次：14
learn rate：0.001
rb均值: -0.4069578648,rb方差:5.367347717285
rb均值: -1.9662336111,rb方差:6.778992176056
rb均值: 1.2839596272,rb方差:9.508784294128
rb均值: 3.3430559635,rb方差:13.487200737000
rb均值: -1.7023583651,rb方差:11.101929664612
rb均值: 1.0313628912,rb方差:8.896734237671
rb均值: 1.1472668648,rb方差:5.608257293701
rb均值: -0.9595762491,rb方差:6.532460689545
rb均值: 6.2563285828,rb方差:6.715904712677
rb均值: -2.3356299400,rb方差:7.490069389343
rb均值: -0.3747546971,rb方差:9.384812355042
rb均值: -0.7444572449,rb方差:5.969283103943
rb均值: -0.2474219054,rb方差:5.143793106079
rb均值: -8.6168422699,rb方差:6.658757209778
rb均值: 0.1427542865,rb方差:6.845678806305
rb均值: -2.8374352455,rb方差:5.146701335907
rb均值: 0.0152288666,rb方差:6.868066787720
时间: 33769.097656 ms
train Classification result: 85.26% ok (used 49984 images)
时间: 2486.777100 ms
Test Classification result: 10.00% ok (used 9984 images)
轮次：15
learn rate：0.0001
rb均值: -0.4994803071,rb方差:5.640581130981
rb均值: -1.8807585239,rb方差:7.053150177002
rb均值: 1.3576096296,rb方差:9.930914878845
rb均值: 3.4381208420,rb方差:14.269177436829
rb均值: -1.7235798836,rb方差:11.541169166565
rb均值: 0.9123242497,rb方差:9.328988075256
rb均值: 1.5630474091,rb方差:6.013020515442
rb均值: -0.7024898529,rb方差:7.059567451477
rb均值: 6.6749691963,rb方差:7.149848461151
rb均值: -2.1049180031,rb方差:7.639552593231
rb均值: -0.2678545713,rb方差:9.236975669861
rb均值: -0.7875890136,rb方差:6.135538578033
rb均值: -0.2510080934,rb方差:5.122716426849
rb均值: -1.5197883844,rb方差:5.089890480042
rb均值: -8.9060163498,rb方差:6.885478973389
rb均值: -0.9661549926,rb方差:7.333233833313
rb均值: -3.0932664871,rb方差:5.474475860596
rb均值: 0.0489594080,rb方差:6.819167137146
时间: 33589.511719 ms
train Classification result: 91.10% ok (used 49984 images)
时间: 2526.964111 ms
Test Classification result: 79.98% ok (used 9984 images)
轮次：16
learn rate：0.0001
rb均值: -0.5122302771,rb方差:5.807245731354
rb均值: -1.8645973206,rb方差:7.225339889526
rb均值: 1.3521035910,rb方差:10.015411376953
rb均值: 3.4389827251,rb方差:14.315127372742
rb均值: -1.7367321253,rb方差:11.493885993958
rb均值: 0.8520852923,rb方差:9.209759712219
rb均值: 1.4376308918,rb方差:6.131165981293
rb均值: -0.6479660869,rb方差:7.213823318481
rb均值: 6.6936435699,rb方差:7.315595149994
rb均值: -2.1330022812,rb方差:7.546449184418
rb均值: -0.3127706647,rb方差:9.180306434631
rb均值: -0.7672121525,rb方差:6.255437374115
rb均值: -0.1950288415,rb方差:5.177592277527
rb均值: -1.1415898800,rb方差:5.044437408447
rb均值: -8.8101968765,rb方差:6.675973415375
rb均值: -1.0850901604,rb方差:7.491862297058
rb均值: -3.2881112099,rb方差:5.660645961761
rb均值: 0.0450579412,rb方差:6.985112667084
时间: 33676.761719 ms
train Classification result: 93.46% ok (used 49984 images)
时间: 2534.864014 ms
Test Classification result: 80.01% ok (used 9984 images)
轮次：17
learn rate：1e-05
rb均值: -0.5019329190,rb方差:5.819442272186
rb均值: -1.8735647202,rb方差:7.253060817719
rb均值: 1.3504428864,rb方差:10.034479141235
rb均值: 3.4375467300,rb方差:14.326591491699
rb均值: -1.7355362177,rb方差:11.542701721191
rb均值: 0.8355134130,rb方差:9.274394035339
rb均值: 1.4318505526,rb方差:6.130095005035
rb均值: -0.6910514235,rb方差:7.220703601837
rb均值: 6.7040076256,rb方差:7.342965126038
rb均值: -2.1521174908,rb方差:7.564808845520
rb均值: -0.3168754578,rb方差:9.252548217773
rb均值: -0.7701565027,rb方差:6.306058406830
rb均值: -0.1903188229,rb方差:5.197197437286
rb均值: -1.1765931845,rb方差:5.056819438934
rb均值: -8.8074817657,rb方差:6.666650772095
rb均值: -1.1492933035,rb方差:7.548652648926
rb均值: -3.2397632599,rb方差:5.668169021606
rb均值: 0.0468210317,rb方差:7.001743793488
时间: 33673.437500 ms
train Classification result: 94.50% ok (used 49984 images)
时间: 2546.959961 ms
Test Classification result: 80.26% ok (used 9984 images)
轮次：18
learn rate：1e-05
rb均值: -0.4917011857,rb方差:5.819192886353
rb均值: -1.8670983315,rb方差:7.235083103180
rb均值: 1.3480359316,rb方差:10.047347068787
rb均值: 3.4388384819,rb方差:14.359600067139
rb均值: -1.7285544872,rb方差:11.586200714111
rb均值: 0.8305882215,rb方差:9.338629722595
rb均值: 1.4212019444,rb方差:6.142400741577
rb均值: -0.6924264431,rb方差:7.218382835388
rb均值: 6.6812014580,rb方差:7.354331016541
rb均值: -2.1592683792,rb方差:7.592640399933
rb均值: -0.3135059178,rb方差:9.256781578064
rb均值: -0.7741191983,rb方差:6.282418251038
rb均值: -0.1895552278,rb方差:5.204569816589
rb均值: -1.1736658812,rb方差:5.028577327728
rb均值: -8.8007488251,rb方差:6.663414001465
rb均值: -1.1560802460,rb方差:7.551405429840
rb均值: -3.2438018322,rb方差:5.676767826080
rb均值: 0.0486167297,rb方差:7.002242088318
时间: 33739.070312 ms
train Classification result: 94.69% ok (used 49984 images)
时间: 2517.916992 ms
Test Classification result: 80.24% ok (used 9984 images)
但是训练时间从28秒上升到33秒！
估计实线卷积核使用1*1和3*3，降低dropout，估计也能上80分！不行！
刚试过！
想说的还有上升空间的一个版本（也上了80分）：
实线卷积核使用1*1和3*3，dropout=0.5，一轮21秒，够快！
使用标签平滑！使用两次虚线残差！上面都是三次虚线残差！
这个实线残差使用1*1卷积和3*3卷积！
//resnet18//resnet18//resnet18
layers.emplace_back(std::make_shared<Conv2D>(cudnn, batch, 5, 32, 32, 32, 3, 1, 1));
//第一残差开始
layers.emplace_back(std::make_shared<residualExt2>(cudnn, batch, 32, 32, 32));
layers.emplace_back(std::make_shared<ReLU>(cudnn, batch, 32, 32, 32));
//第二残差开始
layers.emplace_back(std::make_shared<Conv2D>(cudnn, batch, 32, 64, 32, 32, 3, 1, 1));
layers.emplace_back(std::make_shared<residualExt2>(cudnn, batch, 64, 32, 32));
layers.emplace_back(std::make_shared<ReLU>(cudnn, batch, 64, 32, 32));
//第三残差开始，虚线
//xuxianResidual
layers.emplace_back(std::make_shared<xuxianResidual2>(cudnn, batch, 64, 32, 32));//out->batch, 128, 16, 16
layers.emplace_back(std::make_shared<ReLU>(cudnn, batch, 128, 16, 16));
layers.emplace_back(std::make_shared<residualExt2>(cudnn, batch, 128, 16, 16));
layers.emplace_back(std::make_shared<ReLU>(cudnn, batch, 128, 16, 16));
layers.emplace_back(std::make_shared<xuxianResidual2>(cudnn, batch, 128, 16, 16));//out->batch, 256, 8, 8
layers.emplace_back(std::make_shared<ReLU>(cudnn, batch, 256, 8, 8));
layers.emplace_back(std::make_shared<residualExt2>(cudnn, batch, 256, 8, 8));
layers.emplace_back(std::make_shared<ReLU>(cudnn, batch, 256, 8, 8));
//砍掉一虚线一实线残差块
layers.emplace_back(std::make_shared<AvgPool2D>(cudnn, batch, 256, 8,8, 2, 2, 0, 2));
layers.emplace_back(std::make_shared<Linear>(cublas, batch, 256 * 16, 384));
layers.emplace_back(std::make_shared<BN>(cudnn, batch, 384, 1, 1));
layers.emplace_back(std::make_shared<ReLU>(cudnn, batch, 384, 1, 1));
layers.emplace_back(std::make_shared<Dropout>(cudnn, batch, 384, 1, 1));//=0.5
layers.emplace_back(std::make_shared<Linear>(cublas, batch, 384, 10));
lr从0.001开始，训练大于等于65开始用lr=0.0001；后面上88，执行两次lr=0.0001；再执行两次lr=0.00001；再执行两次lr=0.000001；再执行两次lr=0.0000001；结束！
轮次：18
learn rate：0.001
时间: 21690.460938 ms
train Classification result: 67.15% ok (used 49984 images)
时间: 1916.250000 ms
Test Classification result: 9.99% ok (used 9984 images)
轮次：19
learn rate：0.0001
时间: 21763.904297 ms
train Classification result: 73.80% ok (used 49984 images)
时间: 1865.605957 ms
Test Classification result: 10.00% ok (used 9984 images)
轮次：20
learn rate：0.0001
时间: 21664.189453 ms
train Classification result: 75.36% ok (used 49984 images)
时间: 1881.493042 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：21
learn rate：0.0001
时间: 21763.871094 ms
train Classification result: 76.44% ok (used 49984 images)
时间: 1880.659058 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：22
learn rate：0.0001
时间: 21713.888672 ms
train Classification result: 76.95% ok (used 49984 images)
时间: 1874.404053 ms
Test Classification result: 9.99% ok (used 9984 images)
轮次：23
learn rate：0.0001
时间: 21694.732422 ms
train Classification result: 77.42% ok (used 49984 images)
时间: 1880.053955 ms
Test Classification result: 10.02% ok (used 9984 images)
轮次：24
learn rate：0.0001
时间: 21805.474609 ms
train Classification result: 77.94% ok (used 49984 images)
时间: 1878.199951 ms
Test Classification result: 10.00% ok (used 9984 images)
轮次：25
learn rate：0.0001
时间: 21697.507812 ms
train Classification result: 78.43% ok (used 49984 images)
时间: 1863.810059 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：26
learn rate：0.0001
时间: 21790.343750 ms
train Classification result: 78.83% ok (used 49984 images)
时间: 1859.404053 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：27
learn rate：0.0001
时间: 21748.654297 ms
train Classification result: 79.09% ok (used 49984 images)
时间: 1873.607056 ms
Test Classification result: 9.99% ok (used 9984 images)
轮次：28
learn rate：0.0001
时间: 21693.824219 ms
train Classification result: 79.55% ok (used 49984 images)
时间: 1990.251953 ms
Test Classification result: 10.00% ok (used 9984 images)
轮次：29
learn rate：0.0001
时间: 21782.923828 ms
train Classification result: 79.93% ok (used 49984 images)
时间: 1878.133057 ms
Test Classification result: 10.00% ok (used 9984 images)
轮次：30
learn rate：0.0001
时间: 21679.824219 ms
train Classification result: 80.45% ok (used 49984 images)
时间: 1863.098999 ms
Test Classification result: 9.99% ok (used 9984 images)
轮次：31
learn rate：0.0001
时间: 21760.689453 ms
train Classification result: 80.70% ok (used 49984 images)
时间: 1876.437012 ms
Test Classification result: 9.99% ok (used 9984 images)
轮次：32
learn rate：0.0001
时间: 21706.023438 ms
train Classification result: 81.02% ok (used 49984 images)
时间: 1856.899048 ms
Test Classification result: 9.98% ok (used 9984 images)
轮次：33
learn rate：0.0001
时间: 21718.371094 ms
train Classification result: 81.39% ok (used 49984 images)
时间: 1853.427979 ms
Test Classification result: 10.00% ok (used 9984 images)
轮次：34
learn rate：0.0001
时间: 21726.279297 ms
train Classification result: 81.90% ok (used 49984 images)
时间: 1890.915039 ms
Test Classification result: 10.02% ok (used 9984 images)
轮次：35
learn rate：0.0001
时间: 21721.281250 ms
train Classification result: 81.97% ok (used 49984 images)
时间: 1918.589966 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：36
learn rate：0.0001
时间: 21683.271484 ms
train Classification result: 82.42% ok (used 49984 images)
时间: 1872.499023 ms
Test Classification result: 10.00% ok (used 9984 images)
轮次：37
learn rate：0.0001
时间: 21633.812500 ms
train Classification result: 82.57% ok (used 49984 images)
时间: 1866.302979 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：38
learn rate：0.0001
时间: 21706.794922 ms
train Classification result: 83.10% ok (used 49984 images)
时间: 1862.642944 ms
Test Classification result: 10.02% ok (used 9984 images)
轮次：39
learn rate：0.0001
时间: 21707.251953 ms
train Classification result: 83.38% ok (used 49984 images)
时间: 1888.725952 ms
Test Classification result: 10.02% ok (used 9984 images)
轮次：40
learn rate：0.0001
时间: 21767.412109 ms
train Classification result: 83.80% ok (used 49984 images)
时间: 1851.958008 ms
Test Classification result: 10.00% ok (used 9984 images)
轮次：41
learn rate：0.0001
时间: 21765.160156 ms
train Classification result: 84.07% ok (used 49984 images)
时间: 1851.100952 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：42
learn rate：0.0001
时间: 21773.371094 ms
train Classification result: 84.58% ok (used 49984 images)
时间: 1877.822998 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：43
learn rate：0.0001
时间: 21782.539062 ms
train Classification result: 84.97% ok (used 49984 images)
时间: 1908.098999 ms
Test Classification result: 10.02% ok (used 9984 images)
轮次：44
learn rate：0.0001
时间: 22225.150391 ms
train Classification result: 85.17% ok (used 49984 images)
时间: 1937.349976 ms
Test Classification result: 10.02% ok (used 9984 images)
轮次：45
learn rate：0.0001
时间: 21843.072266 ms
train Classification result: 85.40% ok (used 49984 images)
时间: 1929.135010 ms
Test Classification result: 9.99% ok (used 9984 images)
轮次：46
learn rate：0.0001
时间: 22025.341797 ms
train Classification result: 85.95% ok (used 49984 images)
时间: 1899.963013 ms
Test Classification result: 9.99% ok (used 9984 images)
轮次：47
learn rate：0.0001
时间: 21600.160156 ms
train Classification result: 86.08% ok (used 49984 images)
时间: 1870.935059 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：48
learn rate：0.0001
时间: 21989.687500 ms
train Classification result: 86.55% ok (used 49984 images)
时间: 1927.626953 ms
Test Classification result: 10.02% ok (used 9984 images)
轮次：49
learn rate：0.0001
时间: 22005.525391 ms
train Classification result: 86.81% ok (used 49984 images)
时间: 1912.337036 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：50
learn rate：0.0001
时间: 22061.476562 ms
train Classification result: 87.27% ok (used 49984 images)
时间: 1894.775024 ms
Test Classification result: 10.00% ok (used 9984 images)
轮次：51
learn rate：0.0001
时间: 21988.039062 ms
train Classification result: 87.46% ok (used 49984 images)
时间: 1954.899048 ms
Test Classification result: 9.98% ok (used 9984 images)
轮次：52
learn rate：0.0001
时间: 21883.779297 ms
train Classification result: 87.68% ok (used 49984 images)
时间: 1941.171997 ms
Test Classification result: 10.02% ok (used 9984 images)
轮次：53
learn rate：0.0001
时间: 21871.496094 ms
train Classification result: 88.19% ok (used 49984 images)
时间: 1913.165039 ms
Test Classification result: 10.00% ok (used 9984 images)
轮次：54
learn rate：0.0001
时间: 21995.228516 ms
train Classification result: 88.42% ok (used 49984 images)
时间: 1964.012939 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：55
learn rate：0.0001
时间: 22051.152344 ms
train Classification result: 88.89% ok (used 49984 images)
时间: 2000.795044 ms
Test Classification result: 10.01% ok (used 9984 images)
轮次：56
learn rate：1e-05
时间: 21906.990234 ms
train Classification result: 89.92% ok (used 49984 images)
时间: 1923.550049 ms
Test Classification result: 79.77% ok (used 9984 images)
轮次：57
learn rate：1e-05
时间: 21963.455078 ms
train Classification result: 90.37% ok (used 49984 images)
时间: 1962.576050 ms
Test Classification result: 80.05% ok (used 9984 images)
轮次：58
learn rate：1e-06
时间: 21678.431641 ms
train Classification result: 90.61% ok (used 49984 images)
时间: 1926.390991 ms
Test Classification result: 80.03% ok (used 9984 images)
轮次：59
learn rate：1e-06
时间: 21751.011719 ms
train Classification result: 90.58% ok (used 49984 images)
时间: 1873.609985 ms
Test Classification result: 80.03% ok (used 9984 images)
轮次：60
learn rate：1e-07
时间: 21905.980469 ms
train Classification result: 90.66% ok (used 49984 images)
时间: 1875.582031 ms
Test Classification result: 80.02% ok (used 9984 images)
轮次：61
learn rate：1e-07
时间: 21663.388672 ms
train Classification result: 90.62% ok (used 49984 images)
时间: 1857.135986 ms
Test Classification result: 80.04% ok (used 9984 images)
请按任意键继续. . .
就是执行次数太多，一般30内结束战斗，这都整了61轮，还好21一轮，可以接受！
早早用lr=0.0001，好处多多！
均值和方差也是稳稳地！