[CF-metadata] Handling time when date is "missing"

[Ajay Krishnan - NOAA Affiliate]

Hi All,

I have a user that's converting some IMMA format files to CF compliant
NetCDF files.

The problem is that, we've run into several measurements where just the
hour of measurement has been recorded without the corresponding "date". We
would prefer not to omit these data in the conversion, because they are
considered valid measurements (and play a role in monthly summary
statistics)

How do we represent this in a valid CF NetCDF format since we can't use
_FillValues for 'time'? Any suggestions for handling such special cases?

Thanks,
Ajay

[Dave Allured - NOAA Affiliate]

Ajay,

I think this is an exception to CF. I recommend using _FillValue or
missing_value on the time coordinate. Document this in a comment attribute
on the time coordinate variable.

Also document this somehow in another global attribute that explains you
made this exception to the CF conventions. Follow CF conventions in all
other regards.

Then, try to remember to warn people about this when you distribute the
data. CF compliant time coordinates are fundamental to many application
programs, and I expect they will choke or introduce subtle errors if
missing values are in there. So users will need to provide special
handling for such files. HTH.

--Dave
(Please reply to list only)


On Tue, Oct 25, 2016 at 12:07 PM, Ajay Krishnan - NOAA Affiliate <

Post by Ajay Krishnan - NOAA Affiliate
Hi All,
I have a user that's converting some IMMA format files to CF compliant
NetCDF files.
The problem is that, we've run into several measurements where just the
hour of measurement has been recorded without the corresponding "date". We
would prefer not to omit these data in the conversion, because they are
considered valid measurements (and play a role in monthly summary
statistics)
How do we represent this in a valid CF NetCDF format since we can't use
_FillValues for 'time'? Any suggestions for handling such special cases?
Thanks,
Ajay

[Seth McGinnis]

But then the data is non-compliant, and it sounds like a valid CF
solution is needed.

Two possible solutions come to my mind. The first way would be to store
the undated measurements separately. Record the normal measurements in
the normal way, and then record the undated measurements in a separate
data variable with an index coordinate instead of a time coordinate.

The other way would be not to use time as a coordinate variable at all,
but only as a data variable. Record all the measurements with an index
coordinate instead of a time coordinate. Then define data variables for
year, month, day, and time of measurement, and just fill in what's known
for each one. (It sounds like the month and year are still known even
if the day is not.) This is very similar to the approach taken for
trajectories; see example H.12 in the spec.

Cheers,

--Seth

Post by Dave Allured - NOAA Affiliate
Ajay,
I think this is an exception to CF. I recommend using _FillValue or
missing_value on the time coordinate. Document this in a comment
attribute on the time coordinate variable.
Also document this somehow in another global attribute that explains you
made this exception to the CF conventions. Follow CF conventions in all
other regards.
Then, try to remember to warn people about this when you distribute the
data. CF compliant time coordinates are fundamental to many application
programs, and I expect they will choke or introduce subtle errors if
missing values are in there. So users will need to provide special
handling for such files. HTH.
--Dave
(Please reply to list only)
On Tue, Oct 25, 2016 at 12:07 PM, Ajay Krishnan - NOAA Affiliate
Hi All,
I have a user that's converting some IMMA format files to CF
compliant NetCDF files.
The problem is that, we've run into several measurements where just
the hour of measurement has been recorded without the corresponding
"date". We would prefer not to omit these data in the conversion,
because they are considered valid measurements (and play a role in
monthly summary statistics)
How do we represent this in a valid CF NetCDF format since we can't
use _FillValues for 'time'? Any suggestions for handling such
special cases?
Thanks,
Ajay
_______________________________________________
CF-metadata mailing list
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[Dave Allured - NOAA Affiliate]

Seth, Jim, et al:

Thank you for the constructive alternatives.

--Dave

Post by Seth McGinnis
But then the data is non-compliant, and it sounds like a valid CF
solution is needed.
Two possible solutions come to my mind. The first way would be to store
the undated measurements separately. Record the normal measurements in
the normal way, and then record the undated measurements in a separate
data variable with an index coordinate instead of a time coordinate.
The other way would be not to use time as a coordinate variable at all,
but only as a data variable. Record all the measurements with an index
coordinate instead of a time coordinate. Then define data variables for
year, month, day, and time of measurement, and just fill in what's known
for each one. (It sounds like the month and year are still known even
if the day is not.) This is very similar to the approach taken for
trajectories; see example H.12 in the spec.
Cheers,
--Seth

Post by Dave Allured - NOAA Affiliate
Ajay,
I think this is an exception to CF. I recommend using _FillValue or
missing_value on the time coordinate. Document this in a comment
attribute on the time coordinate variable.
Also document this somehow in another global attribute that explains you
made this exception to the CF conventions. Follow CF conventions in all
other regards.
Then, try to remember to warn people about this when you distribute the
data. CF compliant time coordinates are fundamental to many application
programs, and I expect they will choke or introduce subtle errors if
missing values are in there. So users will need to provide special
handling for such files. HTH.
--Dave
(Please reply to list only)
On Tue, Oct 25, 2016 at 12:07 PM, Ajay Krishnan - NOAA Affiliate
Hi All,
I have a user that's converting some IMMA format files to CF
compliant NetCDF files.
The problem is that, we've run into several measurements where just
the hour of measurement has been recorded without the corresponding
"date". We would prefer not to omit these data in the conversion,
because they are considered valid measurements (and play a role in
monthly summary statistics)
How do we represent this in a valid CF NetCDF format since we can't
use _FillValues for 'time'? Any suggestions for handling such
special cases?
Thanks,
Ajay
_______________________________________________
CF-metadata mailing list
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

_______________________________________________
CF-metadata mailing list
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[Jim Biard]

Ajay,

I'm pretty certain that a file with fill values in a true time
coordinate variable will cause most every software package to have a
fit. It will also likely fail all CF compliance checkers.

Do you have any "alternative" method to determine the rest of the date?
If so, I'd suggest doing the best job you can of reconstructing the full
date/time for the time axis and adding a quality variable that would
mark those time steps as being provisional.

Another approach is to make a time variable that is not a coordinate
variable. You could have something like:

dimension time = UNLIMITED ;

# This variable holds your acquired time stamp information.
#
variable acq_time(time):
:_FillValue = -1.0e20 ;

# This variable holds your notional time coordinate.
#
variable time(time):

variable data(time):


Or, you could dispense with a true time coordinate variable all together.

dimension sample = UNLIMITED ;

# This variable holds your acquired time stamp information.
#
variable acq_time(sample):
:_FillValue = -1.0e20 ;

variable data(sample):
:coordinates = "acq_time" ;


Both of these approaches avoid the problem of having fill values in a
true coordinate variable.

Grace and peace,

Jim

Post by Ajay Krishnan - NOAA Affiliate
Hi All,
I have a user that's converting some IMMA format files to CF compliant
NetCDF files.
The problem is that, we've run into several measurements where just
the hour of measurement has been recorded without the corresponding
"date". We would prefer not to omit these data in the conversion,
because they are considered valid measurements (and play a role in
monthly summary statistics)
How do we represent this in a valid CF NetCDF format since we can't
use _FillValues for 'time'? Any suggestions for handling such special
cases?
Thanks,
Ajay
_______________________________________________
CF-metadata mailing list
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

--
CICS-NC <http://www.cicsnc.org/> Visit us on
Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
*Research Scholar*
Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
North Carolina State University <http://ncsu.edu/>
NOAA National Centers for Environmental Information <http://ncdc.noaa.gov/>
/formerly NOAA’s National Climatic Data Center/
151 Patton Ave, Asheville, NC 28801
e: ***@cicsnc.org <mailto:***@cicsnc.org>
o: [phone]

/Connect with us on Facebook for climate
<https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics
<https://www.facebook.com/NOAANCEIoceangeo> information, and follow us
on Twitter at @NOAANCEIclimate <https://twitter.com/NOAANCEIclimate> and
@NOAANCEIocngeo <https://twitter.com/NOAANCEIocngeo>. /

[Armstrong, Edward M (398G)]

Jay,

You could use the variable time as a single value to establish the time (in complete CF date format) of the first observation

Another multi dimension array can then be used to store the time offset (in hours or seconds etc.) of each measurement from variable time

Or else convert the hourly measurements into a proper CF date format to store in variable time

From: CF-metadata <cf-metadata-***@cgd.ucar.edu<mailto:cf-metadata-***@cgd.ucar.edu>> on behalf of Ajay Krishnan - NOAA Affiliate <***@noaa.gov<mailto:***@noaa.gov>>
Date: Tuesday, October 25, 2016 at 11:07 AM
To: "cf-***@cgd.ucar.edu<mailto:cf-***@cgd.ucar.edu>" <cf-***@cgd.ucar.edu<mailto:cf-***@cgd.ucar.edu>>
Subject: [CF-metadata] Handling time when date is "missing"

Hi All,

I have a user that's converting some IMMA format files to CF compliant NetCDF files.

The problem is that, we've run into several measurements where just the hour of measurement has been recorded without the corresponding "date". We would prefer not to omit these data in the conversion, because they are considered valid measurements (and play a role in monthly summary statistics)

How do we represent this in a valid CF NetCDF format since we can't use _FillValues for 'time'? Any suggestions for handling such special cases?

Thanks,
Ajay