Friday, March 28, 2014

Emissivity

What is emissivity and why is it important? (FAQ - Thermal)

All objects at temperatures above absolute zero emit thermal radiation. However, for any particular wavelength and temperature the amount of thermal radiation emitted depends on the emissivity of the object's surface.

Emissivity is defined as the ratio of the energy radiated from a material's surface to that radiated from a blackbody (a perfect emitter) at the same temperature and wavelength and under the same viewing conditions.

Definition: emissivity = energy from surface / energy from blackbody
Value: [ 0,  1 ]


It is a dimensionless number between 0 (for a perfect reflector) and 1 (for a perfect emitter). The emissivity of a surface depends not only on the material but also on the nature of the surface.

Impact factors:
    Material 
    Nature of surface
    Temperature
    Wavelength
    Angle

( From pic below, it seems that even cloud has higher emissivity than ocean. )

 For example, a clean and polished metal surface will have a low emissivity, whereas a roughened and oxidised metal surface will have a high emissivity. The emissivity also depends on the temperature of the surface as well as wavelength and angle.
Knowledge of surface emissivity is important both for accurate non-contact temperature measurement and for heat transfer calculations. Radiation thermometers detect the thermal radiation emitted by a surface. They are generally calibrated using blackbody reference sources that have an emissivity as close to 1 as makes no practical difference. When viewing 'real' more reflective surfaces, with a lower emissivity, less radiation will be received by the thermometer than from a blackbody at the same temperature and so the surface will appear colder than it is unless the thermometer reading is adjusted to take into account the material surface emissivity. Unfortunately, because the emissivity of a material surface depends on many chemical and physical properties it is often difficult to estimate. It must either be measured or modified in some way, for example by coating the surface with high emissivity black paint, to provide a known emissivity value. The NPL provides a service for measuring the emissivity of samples which is used by customers when they need valid surface temperature measurements or heat transfer calculations.


Thursday, March 27, 2014

Value information in rad data and scene data.

1) Radiance data: 
 nQC:      8
nProf:        16920
nPosScan:      30
nScanLines:     564
freq:
      50.3000      52.8000      53.5960      54.4000      55.5000      57.2900      59.4000
      150.000      183.310      183.310      183.310      19.3500      19.3500      22.2350
      37.0000      37.0000      91.6550      91.6550      63.2832      60.7927      60.7927
      60.7927      60.7927      60.7927
Polarity:
           5           5           5           5           5           2           2           5
           5           5           5           5           4           4           5           4
           4           5           2           2           2           2           2           2
QC: intarr(nfilesRad,nProf,14)
f= where(radd.qc(0,*,*) eq 0)
help, f
F               LONG      = Array[236880]   :  16920 x 14
value : 0 OR  1   ( all zero )
Angle: fltarr(nfilesRad,nProf,nchan), 
F               LONG      = Array[406080]   :   16920 x 24
Value : 53.0 ONLY.
Direc:
f=where (rad.direc eq 0)
help, f
f=where (rad.direc ne 0)
help, f
F               LONG      = Array[8910]
F               LONG      = Array[8010]
Value:  0  OR  1

2) Scene data:
algSN     0
iTyp      0  
nProf      16920
nProfsProcessed   16920
nLev        101
nChan       24
nScanPos    30
nScanLines  564
nAbsorb     2
nParmCLW    100
nParmRain   100
nParmSnow   100
nParmIce    100
nParmGrpl   100
absorbID    1 3
cFreq       
      50.3000      52.8000      53.5960      54.4000      55.5000      57.2900      59.4000
      150.000      183.310      183.310      183.310      19.3500      19.3500      22.2350
      37.0000      37.0000      91.6550      91.6550      63.2832      60.7927      60.7927
      60.7927      60.7927      60.7927
polarity
           5           5           5           5           5           2           2           5
           5           5           5           5           4           4           5           4
           4           5           2           2           2           2           2           2

nQC         1

QC:  intarr(nPrf,nqc), AA intarr(Scene.nqc), readu,iu, qc
f=where(   scene.qc eq 0 )
help, f
F               LONG      = Array[16920]
Value : 0

Thursday, March 20, 2014

IDL operators

IDL Operators

Parentheses
Parentheses are used to group expressions and to enclose function parameter lists.
;Parentheses enclose function argument lists.
SIN(ANG * PI/180.)
;Parentheses specify order of operator evaluation.
(A + 5)/B

Square Brackets
Square brackets are used to create arrays and to enclose array subscripts.
;Use brackets when assigning elements to an array.
ARRAY = [1, 2, 3, 4, 5]
ARRAY = [1, array_1, 3, array_2, 5] 

;Brackets enclose subscripts.
ARRAY[X, Y]
ARRAY(X, Y)   : works in version prior to version 5.0

Mathematical Operators

There are seven basic IDL mathematical operators, described below.

Assignment

A = 32
Compound Assignment Operators ( +=, -=, etc. )
A = A + 100 A += 100

Addition
;Store the sum of 3 and 6 in B. B = 3 + 6

;Store the string value of "John Doe" in B.
B = 'John' + ' ' + 'Doe'

Subtraction and Negation
;Store the value of 5 subtracted from 9 in C. C = 9 - 5
;Change the sign of C. C = -C

Multiplication
; Store the product of 2 and 5 in variable C: C = 2 * 5

Division
; Store the result of 10.0 divided by 3.2 in variable D: D = 10.0/3.2

Exponentiation

The caret (^) is the exponentiation operator. A^B is equal to A raised to the B power.

For real numbers, A^B is evaluated as follows:

    If A is a real number and B is of integer type, repeated multiplication is applied.

    If both A and B are real (non-integer), the formula AB = eBlnA is evaluated.

    A0 is defined as 1.

For complex numbers, A^B is evalutated as follows. The complex number A can be represented as A = a + ib, where a is the real part, and ib is the imaginary part. In polar form, we can represent the complex number as A = reiq = r cosq + ir sinq, where r cosq is the real part, and ir sinq is the imaginary part:

    If A is complex and B is real, the formula AB = (reiq)B = rB (cosBq + isinBq) is evaluated.

    If A is real and B is complex, the formula AB = eBlnA is evaluated.
    If both A and B are complex, the formula AB = eBlnA is evaluated, and the natural logarithm is computed to be ln(A) = ln(reiq) = ln(r) + iq.

Modulo
;Assign the value of 9 modulo 5 (4) to A. A = 9 MOD 5

;Compute angle modulo 2p.
A =(ANGLE + B) MOD (2 * !PI)

Increment/Decrement
Increment and decrement operators can be used, along with a variable, as standalone statements
    A++ or ++A

    A-- or --A

Increment/Decrement Expressions
B = 27
A = B++

In contrast, after executing the following statements, both A and B have a value of 26:

B = 27
A = --B

Minimum and Maximum Operators
The Minimum Operator

The "less than" sign (<) is the IDL minimum operator. The value of "A < B" is equal to the smaller of A or B. For example:

;Set A equal to 3. 
A = 5 < 3

;Set A equal to -6. 
A = 5 < (-6)

;Syntax Error. IDL attempts to perform a subtraction operation if 
;the "-6" is not enclosed in parentheses.
A = 5 < -6

;Set all points in array ARR that are larger than 100 to 100.
ARR = ARR < 100

;Set X to the smallest of the three operands.
X = X0 < X1 < X2


For complex numbers the absolute value (or modulus) is used to determine which value is smaller. If both values have the same magnitude then the first value is returned.

For example:

; Set A equal to 1+2i, since ABS(1+2i) is less than ABS(2-4i)
A = COMPLEX(1,2) < COMPLEX(2,-4)

; Set A equal to 1-2i, since ABS(1-2i) equals ABS(-2+i)
A = COMPLEX(1,-2) < COMPLEX(-2,1)

The Maximum Operator

The "greater than" sign (>) is the IDL maximum operator. "A > B" is equal to the larger of A or B. For example:

;'>' is used to avoid taking the log of zero or negative numbers.
C = ALOG(D > 1E - 6)

;Plot positive points only. Negative points are plotted as zero.
PLOT, ARR > 0

For complex numbers the absolute value (or modulus) is used to determine which value is larger. If both values have the same magnitude then the first value is returned. For example:

; Set A equal to 2-4i, since ABS(2-4i) is greater than ABS(1+2i)
A = COMPLEX(1,2) > COMPLEX(2,-4)

; Set A equal to 1-2i, since ABS(1-2i) equals ABS(-2+i)
A = COMPLEX(1,-2) > COMPLEX(-2,1)

Matrix Multiplication

IDL has two operators used to multiply arrays and matrices.

The # Operator

The # operator computes array elements by multiplying the columns of the first array by the rows of the second array. The second array must have the same number of columns as the first array has rows. The resulting array has the same number of columns as the first array and the same number of rows as the second array.

syntax:  M x N # P x M =  P x N   ( multiple col by row )

The ## Operator  (normal matrix multiplication)

The ## operator does what is commonly referred to as matrix multiplication. It computes array elements by multiplying the rows of the first array by the columns of the second array. The second array must have the same number of rows as the first array has columns. The resulting array has the same number of rows as the first array and the same number of columns as the second array.

For an example illustrating the difference between the two, see Multiplying Arrays.

syntax:  M x N ## N x P =  M x P     ( multiple row by col )


Array Concatenation

The square brackets are used as array concatenation operators.

The expression [A,B] is an array formed by concatenating A and B, which can be scalars or arrays, along the first dimension.

The second and third dimensions can be concatenated by nesting the bracket levels;

[[1,2],[3,4]] is a 2-element by 2-element array with the first row containing 1 and 2 and the second row containing 3 and 4. Operands must have compatible dimensions; all dimensions must be equal except the dimension that is to be concatenated, e.g., [2,INTARR(2,2)] are incompatible. Examples:

;Define C as three-point vector.
C = [-1, 1, -1]

;Add 12 to the end of C.
C = [C, 12]

;Insert 12 at the beginning of C.
C = [12, C]

;Plot ARR2 appended to ARR1.   
PLOT, [ARR1, ARR2]

;Define a 3x3 matrix.
KER = [[1,2,1], [2,4,2], [1,2,1]]

Logical Operators
There are three logical operators in IDL: &&, ||, and ~.

&& (and)

The logical && operator performs the logical short-circuiting "and" operation on two scalars or one-element arrays, returning 1 if both operands are true and 0 if either operand is false.

||  (or)

The logical || operator performs the logical short-circuiting "or" operation on two scalars or one-element arrays, returning 1 if either of the operands is true and 0 if both are false.

~  (not)

The logical ~ operator performs the logical "not" operation on a scalar or array operand. If the operand is a scalar, it returns scalar 1 if the operand is false or scalar 0 if the operand is true. If the operand is an array, it returns an array containing a 1 for each element of the operand array that is false, and a 0 for each element that is true.

Note

Programmers familiar with the C programming language, and the many languages that share its syntax, may expect ~ to perform bitwise negation (1's complement), and for ! to be used for logical negation. This is not the case in IDL: ! is used to reference system variables, the NOT operator performs bitwise negation, and ~ performs logical negation.

When is an Operand True?

When evaluated by a logical operator, an expression is considered to be "true" under the following conditions:

    For numerical operands, if the value is non-zero.
    For string operands, if the value is non-null.
    For heap variables (pointers and object references), if the value is non-null.

Logical Operator Examples

Results of relational expressions can be combined into more complex expressions using the logical operators. Some examples of relational and logical expressions are as follows:

;True if A is between 25 and 50. If A is an array, then the result 
;is an array of zeros and ones.
(A LE 50) && (A GE 25)

;True if A is less than 25 or greater than 50. This is the inverse 
;of the first.
(A GT 50) || (A LT 25)

Bitwise Operators

AND
NOT
OR
XOR

Relational Operators

The IDL relational operators can be used to test the relationship between two arguments. The six relational operators are described in the following table:
EQ
NE
GE
GT
LE
LT

In IDL, the value "true" is represented by the following:

    Any odd, nonzero value for byte, integer, and longword data types

    Any nonzero value for single, double-precision, and the real part of a complex number (the  
          imaginary part is ignored)

    Any non-null string


Using Relational Operators with Arrays

Relational operators can be applied to arrays, and the resulting array of ones and zeroes can be used as an operand. For example, the expression, ARR * (ARR LE 100) is an array equal to ARR except that all points greater than 100 have been reduced to zero. The expression (ARR LE 100) is an array that contains a 1 where the corresponding element of ARR is less than or equal to 100, and zero otherwise. For example, to print the number of positive elements in the array ARR:

PRINT,TOTAL(ARR GT 0)

Using Relational Operators with Infinity and NaN Values

On Windows and Solaris x86 platforms, using relational operators with the values infinity or NaN (Not a Number) causes an "illegal operand" error. The FINITE function's INFINITY and NAN keywords can be used to perform comparisons involving infinity and NaN values. For more information, see FINITE and Special Floating-Point Values.

Conditional Expression

The conditional expression-written with the ternary operator ?:-has the lowest precedence of all the operators. It provides a way to write simple constructions of the IF...THEN...ELSE statement in expression form. In the following example, Z receives the larger of the values contained by A and B:

IF (A GT B) THEN Z = A ELSE Z = B

This statement can be written more concisely using a conditional expression:

Z = (A GT B) ? A : B

The general form of a conditional expression is:

expr1 ? expr2 : expr3

Tuesday, March 18, 2014

Pre-estimation assessment tool development

1. IDL code enhancement: 
The IDL code will take in one run that you're interested at and 
generates all the graphics for assessement. 

1. clear-sky assessment: 
   . map plot:
      - tb 
      - tbf 
      - tb - tbf (diff)
   . scatter plot
      - tb VS tbf 
      - tb - tbf VS latitude (all orbits)
      - tb - tbf VS latitude (ascending orbit)
      - tb - tbf VS latitude (descending orbit)
   -------------------
   Filter to remove : 
   -------------------
      CLW > 0.05
      rwp > 0.01
      gwp > 0.01

   Implementation order: 
   a) use NWP to filter data
   b) use EDR to filter data

2. Cloudy sky (non-precipitation cloud)
   . map plot:
      - tb 
      - tbf 
      - tb - tbf (diff)
   . scatter plot:
      - tb VS tbf 
      - tb - tbf VS CLW 
   ------------------
   Filter to remove: 
   ------------------
      RWP gt 0.01
      GWP gt 0.01
   ------------------
   Filter to keep:
   ------------------
   a) 0.05 < CLW < 0.3 
   b) 0.05 < CLW 

   Implementation order: 
   a) use NWP to filter data
   b) use EDR to filter data

3. Precipitation
   . map plot: 
      - tb 
      - tbf 
      - tb - tbf (diff)
   . scatter plot:
      - tb VS tbf
      - tb - tbf VS RWP
      - tb - tbf VS GWP
      - tb - tbf VS CLW > 0.3
   ------------------
   Filter to keep:
   ------------------
      RWP > 0.01  
      GWP > 0.01
      possible CLW > 0.3 when using NWP 
      
   Implementation order: 
   a) use NWP to filter data
   b) use EDR to filter data


2. Bias calculation (fortran code) 
1. clear-sky bias (all points) 
   . statistical mean bias 
   . historgram 

1. clear-sky bias (ascending mode)
   . statistical mean bias 
   . historgram 

2. clear-sky bias (descending mode)
   . statistical mean bias 
   . historgram 

3. clear-sky bias (ascending mode)
   . statistical mean bias 
   . historgram 

---------------------
 Clear sky filtering 
---------------------
a) based on NWP CLW/GWP (focused on this)
b) based on EDR CLW/RWP/GWP
c) based on NWP,EDR CLW/RWP/GWP

Path: mirs_trunk/src/testbed/biasGenericAndMonit/
File: Calib_generic_rad.f90
It generates two files: 
   a) mean bias from HIST
      biasCorrec_f18_2013_01_20.dat_gfs
   b) standard deviation
      ModelErrFile_f18_2013_01_20.dat_gfs
 
Implementation process order: 
1. Bias generation
/data/home001/dxu/mirs_trunk/src/testbed/biasGenerAndMonit
Calib_generic_rad.f90  (f90 version)
Calib_generic_rad.pro  (idl version)
Input:  
  ???
Output: 
  biasCorrec_f18_2013_01_20.dat_gfs
  ModelErrFile_f18_2013_01_20.dat_gfs

2. Bias and standard deviation location
/data/home001/dxu/mirs_trunk/data/SemiStaticData/biasCorrec
biasCorrec_f18.dat  (static bias) 
biasCorrec_f18_2013_01_20.dat_gfs
ModelErrFile_f18_2013_01_20.dat_gfs

3. Plot the bias as a function of channel and scan position. 
/net/orbit232l/home/pub/kgarrett/mirs_utilities/src/idl/monitorBias
plotBiasAndAvg.pro 
 
 

Here is what Sid suggested to do: 
1. After applying filters, we can reduce standard deviation. 
2. Use that filters above, we look at bias to optimize 
   observation error.
 
File location:  
1. Scene file: 
location: /data/home001/dxu/mirs_trunk/src/testbed/grid
file: nongridMirs_edr.pro
purpose: to plot scene data

location: /data/home001/dxu/mirs_trunk/src/lib_idl
file: io_scene.pro

location: /data/home001/dxu/mirs_trunk/src/lib/io
file: IO_Scene.f90

2. Bias file: 
location: /data/home001/dxu/mirs_trunk/src/testbed/biasGenerAndMonit
file: Calib_generic_rad.f90
file: Calib_generic_rad.pro

location: /data/home001/dxu/mirs_trunk/data/SemiStaticData/biasCorrec
file: biasCorrec_f18.dat  (static bias) 
file: biasCorrec_f18_2013_01_20.dat_gfs
file: ModelErrFile_f18_2013_01_20.dat_gfs

location: /net/orbit232l/home/pub/kgarrett/mirs_utilities/src/idl/monitorBias
file: plotBiasAndAvg.pro
 
Input and Output data location:
Input:
   a) ssmis input data (what format?)
   /data/home001/dxu/archive_data/mirs_test/f18_ssmis/2013-01-20
   31M NPR.TDRN.SC.D13020.S1452.E1637.B1680102.NS

   b) "gfs 6-hr fcst" valid at
   /data/home001/dxu/archive_data/mirs_test/gfs
   3.3M gfs_sfc2013-01-20.t06
    11M gfs_atm2013-01-20.t06

   c) ecmwf analysis
   /data/home001/dxu/archive_data/mirs_test/ecmwf
    24M ecmwf_sfc2013-08-01.t06
    1.8G ecmwf_atm2013-08-01.t06

Output:
   a) Interpolated from GFS 6-hrs fcst valid at
   /data/home001/dxu/mirs_trunk/data/TestbedData/DynamicData/nwp_analys/f18_ssmis/2013-01-20
   53M NWP_GFSN.SC.D13020.S0423.E0604.B1679596.NS.UAS

   b) obs radiance
   /data/home001/dxu/mirs_trunk/data/TestbedData/DynamicData/fmsdr/f18_ssmis/2013-01-20
   5.0M FMSDR_SN.SC.D13020.S0100.E0248.B1679394.NS.UAS

   c) sim radiance
   /data/home001/dxu/mirs_trunk/data/TestbedData/DynamicData/fwd_analys/f18_ssmis/2013-01-20
   4.5M FWD_GFSN.SC.D13020.S0100.E0248.B1679394.NS.UAS

   d) reg data
   /data/home001/dxu/mirs_trunk/data/TestbedData/DynamicData/regress_retr/f18_ssmis/2013-01-20
   57M REGRESSN.SC.D13020.S0100.E0248.B1679394.NS.UAS

   e) bias data
   /data/home001/dxu/mirs_trunk/data/SemiStaticData/biasCorrec
   27K biasCorrec_f18_2013_01_20.dat_gfs
   572 ModelErrFile_f18_2013_01_20.dat_gfs

   f) 1-dvar output
   /data/home001/dxu/mirs_trunk/data/TestbedData/Outputs/edr/f18_ssmis/2013-01-20
   63M EDR_SN.SC.D13020.S0241.E0427.B1679495.NS.UAS.ORB