[GRASS5] Re: v.in.shp on solaris2.7,5.0.0pre3
David D Gray
ddgray at armadce.demon.co.uk
Sat, 09 Feb 2002 03:04:24 +0000
Markus Neteler wrote:
> On Fri, Feb 08, 2002 at 07:10:27PM +0000, David D Gray wrote:
>>John Gillette wrote:
>>>v.in.shp only works for me on the small test files located in
>>>On bigger files, it stops in the middle.
>>Currently v.in.shape is a memory hog, as it builds huge monolithic
>>structures in memory. On some systems this may be critical, and it also
>>depends on the resources available. Can you say - what happens in the
>>middle of an import, when it stops?
> Hi David,
> do you foresee an update to the topology engine to work segmented (in
There are two approaches to the problem of very large files. I am not
quite sure what constitutes `average', `normal' or `professional-level'
specs in workstation terms these days, but there is a limit, as we know,
to the size of files that v.in.shape can import, and this limit is
currently unrealistically severe.
1) There is the question, which really is much more general than the
problems of import/export, of dealing with small areas at a time, in a
way that allows optimal use of resources, but yet is transparent to the
end-user (even the module-developer). I think this is a reasonable
definition of the question of `segmentation'.
2) The problem of adequate use of disk-access where the problem in hand
exceeds the resources of the available memory.
I foresee (but it may be a mirage :) ) that attacks on both of these are
possible for GRASS 5.1. In the case of disk-access, then even for the
stable release, the use of dbm should solve the problem of large file
> At time I have to process a 300000 lines SHAPE file, 1GB is far from
> being sufficient for v.in.shape. I am asking you as we discussed the
> memory problem some time ago.
> I have successfully plugged in the SHAPE into 5.1, but v.build
> eats all memory. Also 5.0 processing is impossible.
The techniques for segmented import would also provide a new means of
building and maintaining topology. So, the problem of making v.build
efficient and effective is largely the same as that of solving the
This will be the next development after the shape/mif import filters are
ready, and it builds on principles already incorporated in these.
Currently I'm concentarting on getting these ready. There will also be -
shortly afterwards - versions for 5.1, where the main difference will be
in importing multiple categories to the new format.