WIP: Major content update

content/posts/2021/assisted-vectorization.md

@@ -0,0 +1,161 @@
+---
+title: "Image Vectorization 🖇🍮"
+date: 2021-12-08T19:26:46+01:00
+draft: false
+toc: true
+tags:
+  - svg
+  - python
+  - code
+  - image
+---
+
+
+Automated vectorization and upscaling is useful in many scenarios particularly
+for drawn art that can be segmented and is generally structured using strokes
+and gradients. Here I will outline a methodology that is based around
+structural analysis and direct regression methods then evaluate error metrics
+for fine-tuning by comparing the output with non-vectorized up-scalers.
+
+
+## Observations
+
+Regarding image segmentation, I suspect the most common approach is directly
+clustering the entire image using the colour and position. In
+most scenarios this feature-space will be separable and is a well understood
+problem statement. There result still poses some issues; first cluster enclosure
+is difficult to resolve (convexity is not guaranteed), second gradient
+components weaken the separability of the data. In addition we may need to
+sub-sample the image since clustering is computationally expensive given the
+mega-samples of image data.
+
+## Outline
+
+0. Coarse Analysis
+    - Cluster based on colour-space and |∇×F| |∇·F| normalized components
+    - Present image colour and segmentation complexity
+    - Partition image in delta space using histogram method
+1. Pre-processing and edge thresholding
+    - Compute the YCbCr equivalent representation
+    - Map the input colour space based on k-means / SVM for maximum cluster separation
+    - Compute edges in images and fit spline segments with grouping
+2. Fine image segmentation
+    - Use edges to initialize segments and segment regions based on colour deltas
+    - Complete segmentation by filling image
+    - Regression to fit colour for each segment
+4. Image restructuring and grouping
+    - Simplify structures by creating region hierarchy
+    - SVG 2.0 supports mesh-gradients / Bezier surface composition
+    - Detect regular patterns with auto correlation / parameter comparison
+3. Error evaluation and recalibration
+    - Use upscaled reference to evaluate error
+    - Identify segments that need to be restructured with more detail
+
+
+## Action Items
+
+ - Colour surface fitting
+    - Given a set of samples, fit an appropriate colour gradient
+ - Image normalization and pre-processing
+ - Edge composition and image segmentation
+    - Define a model for segments of the image
+    - Define a model for closed and open edges
+    - Evaluate segment coverage of the image
+ - Heuristics for image hierarchy and optimizations
+
+
+## Pre-processing Engine
+
+Currently histogram binning has proven to be very effective for generating
+initial clusters since it scales well with image size. This allows us to
+quickly gauge the complexity of an image in terms of content separability.
+These regions will be used to partition the image as edges are extracted
+before more accurate mapping is performed.
+
+The main challenge here is colour gradients that washout what should be
+obvious centroids. Analysing the samples in batches can prevent this to some
+extent but a more robust approach is needed. We could use a localized grouping
+technique but accuracy may need be that important for this pre-processing step.
+Another technique is that the image can first be clustered using the histogram
+of derivative components followed by sub-classing a histogram for each gradient
+cluster.
+
+This idea for histogram-binning is surprisingly efficiently for artificial
+images where the colour pallet is rich in features. A binary search for
+parameterizing the local maxima detection will very quickly segment a wide
+variety of images into 10 - 30 classifications.
+
+## Edge extraction
+
+At some point we will want to construct a model representing regions and shapes.
+The principle component here is identifying edges segmenting the image. Edge
+detection is relatively strait-forward as we only need look for extrema in the
+derivative components. In most scenarios this is actually quite noisy and
+it is not obvious how we should threshold for what is and what is not an edge.
+
+Here we use the histogram-based clustering result for edge detection region to
+region transitions are discretized and no adaptive threshold is required.
+There will unavoidably be noisy regions where we see this boundary being spread
+out or possibly just a select few pixels appearing in some sub-section due to
+the clustering process. This can mostly be removed with a median filter if
+necessary.
+
+If the initial segmentation are generated based on k-means in the colour space,
+two families of edges will be detected along segments: hard and soft edges.
+Hard edges will correspond to the intended edges seen in the image where as
+soft edges will arise due to the clustering technique. We can classify these
+two families by looking at the norm of the derivative component along such an
+edge. There will be more than one way to asses the correctness here but the
+significance here is that soft edges present boundary conditions during colour
+mapping while hard edges do not. Otherwise visual artefacts will arise are the
+interface of two segments that originally was a smooth transition.
+
+
+## Structural Variability
+
+While we are targeting a-typical images for vectorization it is obvious that
+'sharpness' in the final result depends on a subjective style that is difficult
+to assess in terms of simple regress or interpolation. This problem however is
+central to upscaling algorithms so the methodology here will be that a external
+upscaling tool will guide the vectorization process. For example vectorizing
+a pixel-art image yields better results 'nearest-neighbour' methods opposed to
+Lanczos resampling.
+
+
+## Regression over SVG colour surfaces
+
+The SVG standard supports three methods for specifying colour profiles or
+gradients: Flat, Linear, Radial. There are more advanced mechanisms through
+embedding or meshing multiple components the aforementioned three readily
+allow us to fit a first order colour contour through linear
+regression. This will be our first objective for parameterizing
+the colour for segments in our image. Another thing to note is that the gradient
+can be transformed after being parameterized. This means that the a circular
+gradient can be transformed to realize a family elliptical gradients.
+
+Obviously this will not accommodate all colour contours that we find in images
+but in such scenarios we may adopt piece-wise approximations or more accurate
+masking of each component using the alpha channel. At some point we should
+also be able to resolve mixtures and decompose the contour from a non-linear
+or higher-order surface into multiple simpler contours. Again note that support
+for advanced colour profiles is not well supported so composition through
+these basic elements will yield the best support.
+
+Using linear regression here with a second order polynomial kernel is a very
+efficient method for directly quantifying the focal point of the colour
+gradient if there is one.
+
+
+## Contour estimation
+
+My initial attempt to estimate the contour given a set of points was based on
+using the convex-hull and recursively enclosing the outline in more detail
+by interpolating in between the current outline and finding the closest point
+orthogonal to the outline. This result yields a fast approximation of the
+enclosing outline without many requirements on the set of points other than
+having a dense outline. The drawback was that if it difficult to detect
+incorrect interpolation and only resolves the outline with pixel-level
+precision. If we pre-process the collection of points such that they
+represent detected edges at sub-pixel resolution the later draw-back can be
+addressed. Correctness or hypothesis testing could yield a more robust result
+at the cost of increased complexity.