Opened 2 months ago

Last modified 6 weeks ago

#1597 new enhancement

CONCAT should be supported in WCPS?

Reported by: dmisev Owned by:
Priority: major Milestone: Future
Component: petascope Version: development
Keywords: Cc: pbaumann, bphamhuu, vmerticariu
Complexity: Medium

Description


Change History (9)

comment:1 Changed 6 weeks ago by bphamhuu

should I do it in Petascope?

comment:2 Changed 6 weeks ago by dmisev

ping, should we implement CONCAT support in WCPS?

comment:3 Changed 6 weeks ago by vmerticariu

Pros: useful for creating larger coverages from smaller parts, already implemented in rasql.

Cons: not part of the WCPS standard.

If we go for the implementation we should follow the rasql model:

concat array1 with array 2 with ... with arrayN along dimension

For WCPS we could have:

concat coverage1 with coverage2 with ... with coverageN along Long

What we need to enforce:

  • all involved coverages must have the same crs
  • all involved coverages must have the same axis order (otherwise we can't concat along 1 axis)
  • all involved coverages must have the same resolutions
  • all involved coverages must have the same range fields
  • probably there must be some restriction on the coefficients as well

The result coverage should have:

  • the domain set resulting from concatenation: usually for axis i that is (min(lowerBound_axis_i_coverage1, lowerBound_axis_i_coverage2, ..., lowerBound_axis_i_coverageN) : max(upperBound_axis_i_coverage1, ..., upperBound_axis_i_coverageN)). When done along an irregular axis, the coefficients must be added to the result axis after being translated to the new axis origin.
  • the range fields of the first coverage (not sure what to to with null values though)
  • the extra metadata of the first coverage (or we could concatenate all, or even make something custom)
  • the range set is resulting from the concat rasql

We will probably need this in the federation as well, when we will lift it at WCPS level. However I would only go for it once we have a clear need, or if Bang is running out of tasks :)

comment:4 Changed 6 weeks ago by bphamhuu

thanks Vlad for comment, I'm running out of task now. I can see the tricky problem from the coverage's metadata in WCPS when concatenating along the irregular axis, for example: a 3D coverage A (time (2008-08-01:2008-08-10), detail (2008-08-01, 2008-08-03, 2008-08-05, 2008-08-10), lat, long) concatenates with a subset of coverage A (time (2008-08-01:2008-08-05)).

So, I'm not sure the result of time axis should be what? I think it can use the offset vector by default is 1 (day for ansidate, second for unixtime), then it will be: 2008-08-01:2008-08-10,2008-08-11,2008-08-13,2008-08-15.

the range fields of the first coverage (not sure what to to with null values though)

It can select the null values which are shared by all participating coverages.

the extra metadata of the first coverage (or we could concatenate all, or even make something custom)

It can simply concatenate the extra metadata from participating coverages.

The need for this operation in Petascope can be recalled from Simone's email to dev mail list long time ago.

I'm interested in to edit / run one single WCPS query to put "RGB query" 
+ "False Color" + "False color - cloud masked" maps one close to the 
other (as per the attached png example).
If I'm not wrong the concatenate function exists at rasql level: does 
the concatenation function exist also at petascope level?

comment:5 follow-up: Changed 6 weeks ago by pbaumann

I agree, there are several "spicy" problems coming specifically with coverages. Including: there might be a change in the coverage/axis grid type through concatenation, typically I guess a generalization from regular to irregular. As always, let's have a clear spec before we start.

comment:6 in reply to: ↑ 5 Changed 6 weeks ago by bphamhuu

Replying to pbaumann:

I agree, there are several "spicy" problems coming specifically with coverages. Including: there might be a change in the coverage/axis grid type through concatenation, typically I guess a generalization from regular to irregular. As always, let's have a clear spec before we start.

It is clear for me to do this ticket, except the case with irregular axis. I'm waiting for Vlad's comment in http://rasdaman.org/ticket/1597#comment:4, if he sees another significant problem, then, this ticket will be moved to future.

comment:7 Changed 6 weeks ago by vmerticariu

I would say that you can't concatenate geographically overlapping axes. This happens in the regular case as well: the result of concatenating Lat(0:10) with Lat(12:15) would be Lat(0:15). But what would the result of concatenating Lat(0:10) with Lat(0:5)?

So I believe this is an important distinction that we must make at petascope level, as opposed to array level.

For irregular axes it would be the same, if the axes don't overlap the result axis has the union between the direct positions of the axes that were concatenated.

I'm not sure how this affects our use cases though.

comment:8 Changed 6 weeks ago by bphamhuu

I would say that you can't concatenate geographically overlapping axes. This happens in the regular case as well: the result of concatenating Lat(0:10) with Lat(12:15) would be Lat(0:15). But what would the result of concatenating Lat(0:10) with Lat(0:5)? 

It will be of your formula

the domain set resulting from concatenation: usually for axis i that is (min(lowerBound_axis_i_coverage1, lowerBound_axis_i_coverage2, ..., lowerBound_axis_i_coverageN) : max(upperBound_axis_i_coverage1, ..., upperBound_axis_i_coverageN)). 

Lat(0:10) concatenates with Lat(0:5) is Lat(0:10), the number of grid pixels and off setvector for Lat is changed accordingly. For another example, Lat(0:10) concatenates Lat(20:30), result is Lat(0:30).

I don't see it is possible in this case, too if not follow a convention rule.

comment:9 Changed 6 weeks ago by dmisev

  • Milestone changed from 9.5 to Future

Guys I'd suggest that we put this on hold until some use case pops up. I had some need for concat while formulating queries in testbed 13, but unfortunately I didn't write it in the ticket and can't remember now.
So let's just postpone it for now.

Note: See TracTickets for help on using tickets.