Signal cleaning with missing data, meter reset and wrong value

[Benoit]

Dear SEEQ community,

I’m facing to multiple issue on a meter signal, I would like to clean this signal to analyze it. Unfortunately, I’m not (yet) a SEEQ expert. Could you please give me some tips to help me on this topic?

 

Issue 1 : Signal is missing

The signal is missing due to external factor, I would like to join the last point and the first point with a linear substitute signal.

 

 

Issue 2 : Signal value = 0

The signal = 0 during a short period of time, I would like to not consider this inconsistent value. (Is “$signal.removeOutliers()” the good solution?)

 

Issue 3 : Meter reset to 0

The meter is resetting to 0, I would like to consider the 0 the total value without a resetting to 0.

  

Thank in advance for your help!

[John Cox]

Hello,

Issue 1: Signal is Missing

For this situation, create a new Seeq formula to remove any invalid values in the data gap and then set the max interpolation. The "max interpolation" is the maximum data gap duration over which you want to connect the data points with a line. Set the time parameter (24hr in my example below) to a value longer than any data gaps you want to close.

$signal.validValues().setMaxInterpolation(24h)

 

Issue 2: Signal Value = 0

For this case, $signal.removeOutliers() may be a good solution, if there is only a single data point at 0. 

Something which may work better (and is used more often) is the .remove() function. Create a new Seeq formula and try this:

$signal.remove($signal==0)

Please see additional information in this seeq.org post on the .remove()function: Data Cleansing Tips

 

Issue 3: Meter Reset to 0
There are several ways to handle signal resets in Seeq, depending on the objectives of your analysis. Can you provide more detail about what you want to do for this issue? 
 

[Benoit]

Thank John for your advises!

Regarding issue 3 (meter reset to 0), the objectives of the analysis is to display incremental meter evolution without the effect of the 0 reset (pink dote line in the attached picture).

[John Cox]

Hello,

To do this, you will need to select a time period basis (a Condition in Seeq) over which to compute the signal (and ignore the meter resets). Often, customers want to do this over a daily or weekly time period, but your time basis may be different. 

The following Seeq formula will take the signal, ignore the resets to 0, and display the incremental total for the signal over the time period you specify:
 

// Calculate only positive increases in the signal
// and then compute the running sum over a 
// chosen time period (in this example, daily time periods).
// The time period can be ANY condition the user selects,
// days(), weeks(), months(), etc. It could also be
// a condition for production runs, campaigns, or anything the user defines.
// The condition inside of runningSum() is the basis for the running total of the signal. 
$signal.runningDelta().max(0).runningSum(days('US/Eastern'))

[Benoit]

Hello,

Thank for your tips, it's help me a lot. A still have a questioning regarding linear interpolation and running delta.

$linear_interpolation = $Initial_Meter_signal.removeOutliers().validValues().setMaxInterpolation(3month).toLinear()

The monthly_evolution signal is based on the linear interpolation, unfortunately the runningDelta method looks to not take into account the interpolated value on the signal (flat step in monthly evolution signal). This is creating an error in the monthly calculation if the missing data is occurring at the month end.

Do you have an idea how correct this issue?

$Monthly_evolution = $linear_interpolation.runningDelta().max(0).runningSum(months('Europe/Paris'))

 

$Monthly_consumption = $Monthly_evolution.aggregate(range(), $capsules, durationKey())

$capsules = months("Europe/Paris")

 

[John Cox]

Hello Benoit, 

In looking at your trends and results for Monthly evolution and Monthly consumption, it appears to me that the results look good. I'm not sure I completely understand where the error is coming from. The Monthly consumption totals look reasonable to me for August 2024 and September 2024. 

But, IF during the data gaps you want to assume that the "Initial Meter Signal" is linearly interpolating, you can force a resample of the interpolated signal to a sample frequency that you choose (in my example formula below I chose 2 minutes, but you can choose whatever you want). Please be aware that this approach will fill in "assumed" data values during the data gap time periods based on linearly interpolating during the gaps, so make sure it does what you want. 

$Initial_Meter_signal.removeOutliers().validValues().setMaxInterpolation(3month).toLinear().resample(2min)

As a temporary test, you may want to create a signal with this formula above and trend it with the original meter signal so that you can see what it looks like at the end of a month when there is a data gap.