In one of the previous posts I covered in depth, how I use the term “calibration” in the context of the programmable decade resistor and how the calibration procedure works. Today it’s all about the result.
Also, I’d like to mention the first post in which I defined a rather loose accuracy goal of \(\ll\pm 0.5\% \textrm{ of value} + 0.3 \Omega\). In reality, I expect the device to be much, much better than this. Nevertheless, I will still compare the results to these limits.
The table will specify the following columns:
- Applied value: Value applied to the input for checking the calibration
- Calculated value: Estimation of the actual resistance, based on the previous “adjustment” procedure
- Indicated value: Measurement taken
- Min: Minimum value according to accuracy goals
- Max: Maximum value according to accuracy goals
- Deviation: Deviation between indicated value and applied value in percent
- Result: “PASS”, if within [Min, Max], else “FAIL”
- Deviation Calc. to Ind.: Deviation between Calculated value and Indicated value
First, we start with the four-wire measurement:
| Applied value (\(\Omega\)) | Calculated value (\(\Omega\)) | Indicated value (\(\Omega\)) | Min (\(\Omega\)) | Max (\(\Omega\)) | Deviation (\(\%\)) | Result | Deviation Calc. to Ind. (\(\%\)) |
|---|---|---|---|---|---|---|---|
| 0 | 0.079 | 0.070 | 0.000 | 0.300 | – | PASS | -11.5278 |
| 1 | 1.062 | 1.052 | 0.995 | 1.305 | 5.2383 | PASS | -0.9056 |
| 2 | 2.080 | 2.074 | 1.990 | 2.310 | 3.7170 | PASS | -0.2721 |
| 3 | 3.059 | 3.052 | 2.985 | 3.315 | 1.7271 | PASS | -0.2349 |
| 4 | 4.080 | 4.073 | 3.980 | 4.320 | 1.8132 | PASS | -0.1832 |
| 5 | 5.059 | 5.050 | 4.975 | 5.325 | 1.0041 | PASS | -0.1738 |
| 6 | 6.080 | 6.072 | 5.970 | 6.330 | 1.1936 | PASS | -0.1379 |
| 7 | 7.059 | 7.049 | 6.965 | 7.335 | 0.7020 | PASS | -0.1397 |
| 8 | 8.078 | 8.069 | 7.960 | 8.340 | 0.8623 | PASS | -0.1116 |
| 9 | 9.056 | 9.047 | 8.955 | 9.345 | 0.5178 | PASS | -0.1037 |
| 10 | 10.122 | 10.108 | 9.95 | 10.35 | 1.0764 | PASS | -0.1418 |
| 20 | 20.132 | 20.123 | 19.90 | 20.40 | 0.6164 | PASS | -0.0433 |
| 30 | 30.119 | 30.107 | 29.85 | 30.45 | 0.3557 | PASS | -0.0408 |
| 40 | 40.135 | 40.125 | 39.80 | 40.50 | 0.3127 | PASS | -0.0248 |
| 50 | 50.121 | 50.109 | 49.75 | 50.55 | 0.2172 | PASS | -0.0248 |
| 60 | 60.130 | 60.120 | 59.70 | 60.60 | 0.2001 | PASS | -0.0166 |
| 70 | 70.115 | 70.103 | 69.65 | 70.65 | 0.1478 | PASS | -0.0165 |
| 80 | 80.126 | 80.116 | 79.60 | 80.70 | 0.1446 | PASS | -0.0129 |
| 90 | 90.111 | 90.099 | 89.55 | 90.75 | 0.1101 | PASS | -0.0132 |
| 100 | 100.074 | 100.061 | 99.5 | 100.8 | 0.0609 | PASS | -0.0131 |
| 200 | 200.012 | 200.003 | 199.0 | 201.3 | 0.0014 | PASS | -0.0046 |
| 300 | 299.899 | 299.891 | 298.5 | 301.8 | -0.0363 | PASS | -0.0026 |
| 400 | 399.863 | 399.860 | 398.0 | 402.3 | -0.0351 | PASS | -0.0008 |
| 500 | 499.750 | 499.748 | 497.5 | 502.8 | -0.0504 | PASS | -0.0004 |
| 600 | 599.639 | 599.639 | 597.0 | 603.3 | -0.0602 | PASS | 0.0000 |
| 700 | 699.525 | 699.527 | 696.5 | 703.8 | -0.0676 | PASS | 0.0002 |
| 800 | 799.439 | 799.440 | 796.0 | 804.3 | -0.0700 | PASS | 0.0002 |
| 900 | 900.309 | 900.310 | 895.5 | 904.8 | 0.0345 | PASS | 0.0002 |
| 1k | 1000.200 | 1000.161 | 995 | 1005.3 | 0.0161 | PASS | -0.0039 |
| 2k | 2000.384 | 2000.276 | 1990 | 2010.3 | 0.0138 | PASS | -0.0054 |
| 3k | 3000.274 | 3000.136 | 2985 | 3015.3 | 0.0045 | PASS | -0.0046 |
| 4k | 4000.671 | 4000.605 | 3980 | 4020.3 | 0.0151 | PASS | -0.0016 |
| 5k | 5000.549 | 5000.471 | 4975 | 5025.3 | 0.0094 | PASS | -0.0016 |
| 6k | 6001.078 | 6001.056 | 5970 | 6030.3 | 0.0176 | PASS | -0.0004 |
| 7k | 6999.175 | 6999.094 | 6965 | 7035.3 | -0.0129 | PASS | -0.0012 |
| 8k | 7999.043 | 7998.958 | 7960 | 8040.3 | -0.0130 | PASS | -0.0011 |
| 9k | 8999.147 | 8999.083 | 8955 | 9045.3 | -0.0102 | PASS | -0.0007 |
| 10k | 9999.013 | 9998.949 | 9950 | 10050.3 | -0.0105 | PASS | -0.0006 |
| 20k | 20000.478 | 20001.205 | 19900 | 20100.3 | 0.0060 | PASS | 0.0036 |
| 30k | 29999.948 | 29999.199 | 29850 | 30150.3 | -0.0027 | PASS | -0.0025 |
| 40k | 39999.501 | 39998.617 | 39800 | 40200.3 | -0.0035 | PASS | -0.0022 |
| 50k | 49999.608 | 49998.765 | 49750 | 50250.3 | -0.0025 | PASS | -0.0017 |
| 60k | 60000.084 | 59999.367 | 59700 | 60300.3 | -0.0011 | PASS | -0.0012 |
| 70k | 70000.320 | 70000.372 | 69650 | 70350.3 | 0.0005 | PASS | 0.0001 |
| 80k | 79999.717 | 79999.635 | 79600 | 80400.3 | -0.0005 | PASS | -0.0001 |
| 90k | 90000.133 | 90000.060 | 89550 | 90450.3 | 0.0001 | PASS | -0.0001 |
| 100k | 100000.339 | 99999.994 | 99500 | 100500.3 | 0.0000 | PASS | -0.0003 |
| 200k | 200000.358 | 200001.770 | 199000 | 201000.3 | 0.0009 | PASS | 0.0007 |
| 300k | 300000.139 | 300004.140 | 298500 | 301500.3 | 0.0014 | PASS | 0.0013 |
| 400k | 399999.623 | 400007.160 | 398000 | 402000.3 | 0.0018 | PASS | 0.0019 |
| 500k | 499999.515 | 500009.000 | 497500 | 502500.3 | 0.0018 | PASS | 0.0019 |
| 600k | 600000.241 | 600015.190 | 597000 | 603000.3 | 0.0025 | PASS | 0.0025 |
| 700k | 699999.739 | 700015.190 | 696500 | 703500.3 | 0.0022 | PASS | 0.0022 |
| 800k | 799999.769 | 800017.240 | 796000 | 804000.3 | 0.0022 | PASS | 0.0022 |
| 900k | 900000.234 | 900018.830 | 895500 | 904500.3 | 0.0021 | PASS | 0.0021 |
As expected, the programmable decade resistor stays well within the original spec. At the lower end there is some significant deviation, mostly caused by the relay contact resistances, trace resistance and so on. At the upper end, the performance is fairly impressive and shows how well the algorithm described in this post actually works. Also, the estimation of the actual resistance (that is shown on the display) is very close to the measured value. But what about values that weren’t used in the “adjustment” process? Glad you asked:
| Applied value (\(\Omega\)) | Calculated value (\(\Omega\)) | Indicated value (\(\Omega\)) | Min (\(\Omega\)) | Max (\(\Omega\)) | Deviation (\(\%\)) | Result | Deviation Calc. to Ind. (\(\%\)) |
|---|---|---|---|---|---|---|---|
| 15 | 15.102 | 15.090 | 14.785 | 15.215 | 0.5977 | PASS | -0.0817 |
| 16 | 16.123 | 16.112 | 15.784 | 16.216 | 0.7003 | PASS | -0.0679 |
| 723 | 722.505 | 722.509 | 722.077 | 723.923 | -0.0680 | PASS | 0.0005 |
| 854 | 853.429 | 853.431 | 852.946 | 855.054 | -0.0666 | PASS | 0.0002 |
| 12345 | 12344.545 | 12344.718 | 12332.455 | 12357.545 | -0.0023 | PASS | 0.0014 |
| 34567 | 34566.719 | 34565.325 | 34532.233 | 34601.767 | -0.0048 | PASS | -0.0040 |
| 56432 | 56431.787 | 56430.315 | 56375.368 | 56488.632 | -0.0030 | PASS | -0.0026 |
| 123456 | 123455.769 | 123456.800 | 123332.344 | 123579.656 | 0.0006 | PASS | 0.0008 |
| 456789 | 456788.710 | 456790.540 | 456332.011 | 457245.989 | 0.0003 | PASS | 0.0004 |
This, of course, is only a small sample, but it is in accordance with the results shown above. So I have nothing to complain here. In fact, the calibration would have easily passed the spec of \(\ll\pm 0.1\% \textrm{ of value} + 0.2 \Omega\) for the tested values, however with some questions about repeatability and untested values.
Next up without any further comments is the two-wire measurement:
| Applied value (\(\Omega\)) | Calculated value (\(\Omega\)) | Indicated value (\(\Omega\)) | Min (\(\Omega\)) | Max (\(\Omega\)) | Deviation (\(\%\)) | Result | Deviation Calc. to Ind. (\(\%\)) |
|---|---|---|---|---|---|---|---|
| 0 | 0.087 | 0.103 | 0 | 0.300 | – | PASS | 18.5034 |
| 1 | 1.061 | 1.084 | 0.995 | 1.305 | 8.4460 | PASS | 2.2111 |
| 2 | 2.085 | 2.106 | 1.990 | 2.310 | 5.2880 | PASS | 0.9956 |
| 3 | 3.066 | 3.082 | 2.985 | 3.315 | 2.7392 | PASS | 0.5276 |
| 4 | 4.087 | 4.104 | 3.980 | 4.320 | 2.5987 | PASS | 0.4147 |
| 5 | 5.068 | 5.081 | 4.975 | 5.325 | 1.6152 | PASS | 0.2518 |
| 6 | 6.088 | 6.102 | 5.970 | 6.330 | 1.7009 | PASS | 0.2309 |
| 7 | 7.068 | 7.079 | 6.965 | 7.335 | 1.1332 | PASS | 0.1603 |
| 8 | 8.088 | 8.102 | 7.960 | 8.340 | 1.2708 | PASS | 0.1690 |
| 9 | 9.067 | 9.079 | 8.955 | 9.345 | 0.8744 | PASS | 0.1290 |
| 10 | 10.137 | 10.141 | 9.95 | 10.35 | 1.4102 | PASS | 0.0397 |
| 20 | 20.147 | 20.157 | 19.90 | 20.40 | 0.7845 | PASS | 0.0492 |
| 30 | 30.134 | 30.139 | 29.85 | 30.45 | 0.4650 | PASS | 0.0182 |
| 40 | 40.150 | 40.160 | 39.80 | 40.50 | 0.3993 | PASS | 0.0242 |
| 50 | 50.136 | 50.142 | 49.75 | 50.55 | 0.2849 | PASS | 0.0128 |
| 60 | 60.145 | 60.153 | 59.70 | 60.60 | 0.2551 | PASS | 0.0134 |
| 70 | 70.130 | 70.136 | 69.65 | 70.65 | 0.1943 | PASS | 0.0086 |
| 80 | 80.141 | 80.148 | 79.60 | 80.70 | 0.1855 | PASS | 0.0093 |
| 90 | 90.126 | 90.132 | 89.55 | 90.75 | 0.1461 | PASS | 0.0061 |
| 100 | 100.095 | 100.092 | 99.5 | 100.8 | 0.0922 | PASS | -0.0028 |
| 200 | 200.033 | 200.034 | 199.0 | 201.3 | 0.0168 | PASS | 0.0003 |
| 300 | 299.920 | 299.921 | 298.5 | 301.8 | -0.0262 | PASS | 0.0005 |
| 400 | 399.885 | 399.889 | 398.0 | 402.3 | -0.0277 | PASS | 0.0010 |
| 500 | 499.772 | 499.777 | 497.5 | 502.8 | -0.0446 | PASS | 0.0010 |
| 600 | 599.661 | 599.668 | 597.0 | 603.3 | -0.0553 | PASS | 0.0012 |
| 700 | 699.548 | 699.556 | 696.5 | 703.8 | -0.0635 | PASS | 0.0011 |
| 800 | 799.462 | 799.469 | 796.0 | 804.3 | -0.0663 | PASS | 0.0009 |
| 900 | 900.322 | 900.340 | 895.5 | 904.8 | 0.0377 | PASS | 0.0020 |
| 1k | 1000.230 | 1000.197 | 995.0 | 1005.3 | 0.0197 | PASS | -0.0033 |
| 2k | 2000.415 | 2000.317 | 1990.0 | 2010.3 | 0.0158 | PASS | -0.0049 |
| 3k | 3000.303 | 3000.178 | 2985.0 | 3015.3 | 0.0059 | PASS | -0.0042 |
| 4k | 4000.709 | 4000.642 | 3980.0 | 4020.3 | 0.0161 | PASS | -0.0017 |
| 5k | 5000.584 | 5000.510 | 4975.0 | 5025.3 | 0.0102 | PASS | -0.0015 |
| 6k | 6001.117 | 6001.093 | 5970.0 | 6030.3 | 0.0182 | PASS | -0.0004 |
| 7k | 6999.223 | 6999.121 | 6965.0 | 7035.3 | -0.0126 | PASS | -0.0015 |
| 8k | 7999.090 | 7998.987 | 7960.0 | 8040.3 | -0.0127 | PASS | -0.0013 |
| 9k | 8999.195 | 8999.112 | 8955.0 | 9045.3 | -0.0099 | PASS | -0.0009 |
| 10k | 9999.059 | 9998.980 | 9950 | 10050.3 | -0.0102 | PASS | -0.0008 |
| 20k | 19999.529 | 20000.150 | 19900 | 20100.3 | 0.0007 | PASS | 0.0031 |
| 30k | 30000.022 | 29999.306 | 29850 | 30150.3 | -0.0023 | PASS | -0.0024 |
| 40k | 39999.608 | 39998.717 | 39800 | 40200.3 | -0.0032 | PASS | -0.0022 |
| 50k | 49999.685 | 49998.848 | 49750 | 50250.3 | -0.0023 | PASS | -0.0017 |
| 60k | 60000.198 | 59999.517 | 59700 | 60300.3 | -0.0008 | PASS | -0.0011 |
| 70k | 70000.391 | 70000.476 | 69650 | 70350.3 | 0.0007 | PASS | 0.0001 |
| 80k | 79999.826 | 79999.733 | 79600 | 80400.3 | -0.0003 | PASS | -0.0001 |
| 90k | 90000.329 | 90000.176 | 89550 | 90450.3 | 0.0002 | PASS | -0.0002 |
| 100k | 99999.596 | 99999.369 | 99500 | 100500.3 | -0.0006 | PASS | -0.0002 |
| 200k | 200000.471 | 200001.976 | 199000 | 201000.3 | 0.0010 | PASS | 0.0008 |
| 300k | 300000.285 | 300003.766 | 298500 | 301500.3 | 0.0013 | PASS | 0.0012 |
| 400k | 399999.594 | 400006.496 | 398000 | 402000.3 | 0.0016 | PASS | 0.0017 |
| 500k | 500000.238 | 500008.336 | 497500 | 502500.3 | 0.0017 | PASS | 0.0016 |
| 600k | 600000.085 | 600012.116 | 597000 | 603000.3 | 0.0020 | PASS | 0.0020 |
| 700k | 700000.072 | 700014.896 | 696500 | 703500.3 | 0.0021 | PASS | 0.0021 |
| 800k | 799999.664 | 800016.656 | 796000 | 804000.3 | 0.0021 | PASS | 0.0021 |
| 900k | 899999.706 | 900015.776 | 895500 | 904500.3 | 0.0018 | PASS | 0.0018 |