Source Maps
Parcel utilizes the package @parcel/source-maps
for processing all source maps to ensure performance and reliability when manipulating source maps across plugins and Parcel's core. This library has been written from the ground up in C++ with both source map manipulation and concatenation in mind and gave us a 20x performance improvement over our old solution using Mozilla's source-map
library and some internal utilities. This improvement in performance is mainly due to optimizations in the data structures and the way in which we cache source maps.
¶ How to use the library
To use the library, you start off by creating an instance of the exported SourceMap
class, on which you can call various functions to add and edit source mappings.
Below is an example covering all ways of adding mappings to a SourceMap
instance:
import SourceMap from "@parcel/source-map";
let sourcemap = new SourceMap();
// Each function that adds mappings has optional offset arguments.
// These can be used to offset the generated mappings by a certain amount.
let lineOffset = 0;
let columnOffset = 0;
// Add indexed mappings
// these are mappings that can sometimes be extracted from a library even before they get converted into VLQ Mappings
sourcemap.addIndexedMappings(
[
{
generated: {
// line index starts at 1
line: 1,
// column index starts at 0
column: 4,
},
original: {
// line index starts at 1
line: 1,
// column index starts at 0
column: 4,
},
source: "index.js",
// Name is optional
name: "A",
},
],
lineOffset,
columnOffset
);
// Add raw mappings, this is what would be outputted into a vlq encoded source-map
sourcemap.addRawMappings(
{
file: "min.js",
names: ["bar", "baz", "n"],
sources: ["one.js", "two.js"],
sourceRoot: "/the/root",
mappings:
"CAAC,IAAI,IAAM,SAAUA,GAClB,OAAOC,IAAID;CCDb,IAAI,IAAM,SAAUE,GAClB,OAAOA",
},
lineOffset,
columnOffset
);
// Sourcemaps can be saved as buffers (flatbuffers), this is what we use for caching in Parcel.
// You can instantiate a SourceMap with these buffer values using the `addBufferMappings` function
let originalMapBuffer = new Buffer();
sourcemap.addBufferMappings(originalMapBuffer, lineOffset, columnOffset);
¶ Transformations/Manipulations
If your plugin does any code manipulations, you should ensure that it creates correct mappings to the original source code to guarantee that we still end up creating an accurate source map at the end of the bundling process. You are expected to return a SourceMap
instance at the end of a transform in a Transformer plugin. We also provide the source map from the previous transform to ensure you map to the original source code and not just the output of the previous transform.
The asset
value that gets passed in the parse
, transform
and generate
functions of a transformer plugin contains a function called getMap()
and getMapBuffer()
, these functions can be used to get a SourceMap instance (getMap()
) and the cached SourceMap Buffer (getMapBuffer()
).
You are free to manipulate the sourcemap at any of these steps in the transformer as long as you ensure the sourcemap that gets returned in generate
maps to the original sourcefile correctly.
Below is an example on how to manipulate sourcemaps in a transformer plugin:
import { Transformer } from "@parcel/plugin";
import SourceMap from "@parcel/source-map";
export default new Transformer({
// ...
async generate({ asset, ast, resolve, options }) {
let compilationResult = dummyCompiler(await asset.getAST());
let map = null;
if (compilationResult.map) {
// If the compilationResult returned a map we convert it to a Parcel SourceMap instance
map = new SourceMap();
// The dummy compiler returned a full, encoded sourcemap with vlq mappings
// Some compilers might have the possibility of returning indexedMappings which might improve performance (like Babel does)
// in general each compiler is able to return rawMappings, so it's always a safe bet to use this
map.addRawMappings(compilationResult.map);
// We get the original map buffer from the asset
// to extend our mappings on top of it to ensure we are mapping to the original source
// instead of the previous transformation
let originalMapBuffer = await asset.getMapBuffer();
if (originalMapBuffer) {
// The `extends` function uses the provided map to remap the original source positions of the map it is called on
// So in this case the original source positions of `map` get remapped to the positions in `originalMapBuffer`
map.extends(originalMapBuffer);
}
}
return {
code: compilationResult.code,
// Make sure to return the map
// we need it for concatenating the sourcemaps together in the final bundle's sourcemap
map,
};
},
});
If your compiler supports the option to pass in an existing sourcemap, you can also use that as it could result in more accurate/better sourcemaps than using the method in the previous example.
An example of how this would work:
import { Transformer } from "@parcel/plugin";
import SourceMap from "@parcel/source-map";
export default new Transformer({
// ...
async generate({ asset, ast, resolve, options }) {
// Get the original map from the asset
let originalMap = await asset.getMap();
let compilationResult = dummyCompiler(await asset.getAST(), {
// Pass the VLQ encoded version of the originalMap to the compiler
originalMap: originalMap.toVLQ(),
});
// In this case the compiler is responsible for mapping to the original positions provided in the originalMap
// so we can just convert it to a Parcel SourceMap and return it
let map = new SourceMap();
if (compilationResult.map) {
map.addRawMappings(compilationResult.map);
}
return {
code: compilationResult.code,
map,
};
},
});
¶ Concatenating sourcemaps in Packagers
If you're writing a custom packager, it's your responsibility to concatenate the sourcemaps of all the assets while packaging the assets. This is done by creating a new SourceMap
instance and adding new mappings to it using the addBufferMappings(buffer, lineOffset, columnOffset)
function. lineOffset
should be equal to the line index at which the asset output starts.
Below is an example of how to do this:
import { Packager } from "@parcel/plugin";
import SourceMap from "@parcel/source-map";
export default new Packager({
async package({ bundle, options }) {
// We instantiate the contents variable, which will content a string which represents the entire output bundle
let contents = "";
// We instantiate a new SourceMap to which we'll add all asset maps
let map = new SourceMap();
// This is a queue that reads in all file content and maps and saves them for use in the actual packaging
let queue = new PromiseQueue({ maxConcurrent: 32 });
bundle.traverse((node) => {
if (node.type === "asset") {
queue.add(async () => {
let [code, mapBuffer] = await Promise.all([
node.value.getCode(),
bundle.target.sourceMap && node.value.getMapBuffer(),
]);
return { code, mapBuffer };
});
}
});
let i = 0;
// Process the entire queue...
let results = await queue.run();
// We traverse the bundle and add the contents of each asset to contents and the mapBuffer's to the map
bundle.traverse((node) => {
if (node.type === "asset") {
// Get the data from the queue results
let { code, mapBuffer } = results[i];
// Add the output to the contents
let output = code || "";
contents += output;
// If Parcel requires sourcemaps we add the mapBuffer to the map
if (options.sourceMaps) {
if (mapBuffer) {
// we add the mapBuffer to the map with the lineOffset
// The lineOffset is equal to the line the content of the asset starts at
// which is the same as the contents length before this asset was added
map.addBufferMappings(mapBuffer, lineOffset);
}
// We add the amount of lines of the current asset to the lineOffset
// this way we know the length of `contents` without having to recalculate it each time
lineOffset += countLines(output) + 1;
}
i++;
}
});
// Return the contents and map so Parcel Core can save these to disk or get post-processed by optimizers
return { contents, map };
},
});
¶ Concatenating ASTs
If you're concatenating ASTs instead of source contents you already have the source mappings embedded into the AST which you can use to generate the final sourcemap. You however have to ensure that those mappings stay intact while editing the nodes, sometimes this can be quite challenging if you're doing a lot of modifications.
An example of how this works:
import { Packager } from "@parcel/plugin";
import SourceMap from "@parcel/source-map";
export default new Packager({
async package({ bundle, options }) {
// Do the AST concatenation and return the compiled result
let compilationResult = concatAndCompile(bundle);
// Create the final packaged sourcemap
let map = new SourceMap();
if (compilationResult.map) {
map.addRawMappings(compilationResult.map);
}
// Return the compiled code and map
return {
code: compilationResult.code,
map,
};
},
});
¶ Postprocessing source maps in optimizers
Using source maps in optimizers is identical to how you use it in transformers as you get one file as input and are expected to return that same file as output but optimized.
The only difference with optimizers is that the map is not provided as part of an asset but rather as a separate parameter/option as you can see in the code snippet below. As always, the map is an instance of the SourceMap
class.
// The contents and map are passed separately
async optimize({ bundle, contents, map }) {
return { contents, map }
}
¶ Diagnosing issues
If you encounter incorrect mappings and want to debug these mappings we have built tools that can help you diagnose these issues. By running a specific reporter (@parcel/reporter-sourcemap-visualiser
), Parcel create a sourcemap-info.json
file with all the necessary information to visualize all the mappings and source content.
To enable it, add a custom .parcelrc
:
{
"extends": "@parcel/config-default",
"reporters": ["...", "@parcel/reporter-sourcemap-visualiser"]
}
After the reporter has created the sourcemap-info.json
file, you can upload it to the sourcemap visualizer
¶ @parcel/source-maps
: API
SourceMap source-map/src/SourceMap.js:7
interface SourceMap {
static generateEmptyMap(sourceName: string, sourceContent: string, lineOffset: number): SourceMap,
Params:
sourceName
: path of the source filesourceContent
: content of the source filelineOffset
: an offset that gets added to the sourceLine index of each mapping
addEmptyMap(sourceName: string, sourceContent: string, lineOffset: number): SourceMap,
Params:
sourceName
: path of the source filesourceContent
: content of the source filelineOffset
: an offset that gets added to the sourceLine index of each mapping
addRawMappings(map: VLQMap, lineOffset: number, columnOffset: number): SourceMap,
addBufferMappings(buffer: Buffer, lineOffset: number, columnOffset: number): SourceMap,
Params:
buffer
: the sourcemap buffer that should get appended to this sourcemaplineOffset
: an offset that gets added to the sourceLine index of each mappingcolumnOffset
: an offset that gets added to the sourceColumn index of each mapping
addIndexedMapping(mapping: IndexedMapping<string>, lineOffset?: number, columnOffset?: number): void,
Params:
mapping
: the mapping that should be appended to this sourcemaplineOffset
: an offset that gets added to the sourceLine index of each mappingcolumnOffset
: an offset that gets added to the sourceColumn index of each mapping
addIndexedMappings(mappings: Array<IndexedMapping<string>>, lineOffset?: number, columnOffset?: number): SourceMap,
Note: This is only faster if they generate the serialised map lazily Note: line numbers start at 1 due to mozilla's source-map library
Params:
mappings
: an array of mapping objectslineOffset
: an offset that gets added to the sourceLine index of each mappingcolumnOffset
: an offset that gets added to the sourceColumn index of each mapping
addName(name: string): number,
Params:
name
: the name that should be appended to the names array
addNames(names: Array<string>): Array<number>,
Params:
names
: an array of names to add to the sourcemap
addSource(source: string): number,
Params:
source
: a filepath that should be appended to the sources array
addSources(sources: Array<string>): Array<number>,
Params:
sources
: an array of filepaths which should sbe appended to the sources array
getSourceIndex(source: string): number,
Params:
source
: the filepath of the source file
getSource(index: number): string,
Params:
index
: the index of the source in the sources array
setSourceContent(sourceName: string, sourceContent: string): void,
Params:
sourceName
: the path of the sourceFilesourceContent
: the content of the sourceFile
getSourceContent(sourceName: string): string,
Params:
sourceName
: filename
getNameIndex(name: string): number,
Params:
name
: the name you want to find the index of
getName(index: number): string,
Params:
index
: the index of the name in the names array
indexedMappingToStringMapping(mapping: ?IndexedMapping<number>): ?IndexedMapping<string>,
Note: This is only used internally, should not be used externally and will probably eventually get handled directly in C++ for improved performance
Params:
index
: the Mapping that should get converted to a string-based Mapping
extends(buffer: Buffer): SourceMap,
This works by finding the closest generated mapping in the provided map to original mappings of this map and remapping those to be the original mapping of the provided map.
Params:
buffer
: exported SourceMap as a flatbuffer
getMap(): ParsedMap,
Note: This is a fairly slow operation
findClosestMapping(line: number, column: number): ?IndexedMapping<string>,
Params:
line
: the line in the generated code (starts at 1)column
: the column in the generated code (starts at 0)
toBuffer(): Buffer,
toVLQ(): VLQMap,
delete(): void,
stringify(options: SourceMapStringifyOptions): Promise<string | VLQMap>,
Params:
options
: options used for formatting the serialised map
}
Referenced by:
BaseAsset, BundleResult, GenerateOutput, MutableAsset, Optimizer, Packager, TransformerResultMappingPosition source-map/src/types.js:2
type MappingPosition = {|
line: number,
column: number,
|}
Referenced by:
IndexedMappingIndexedMapping source-map/src/types.js:7
type IndexedMapping<T> = {
generated: MappingPosition,
original?: MappingPosition,
source?: T,
name?: T,
}
Referenced by:
ParsedMap, SourceMapParsedMap source-map/src/types.js:15
type ParsedMap = {|
sources: Array<string>,
names: Array<string>,
mappings: Array<IndexedMapping<number>>,
sourcesContent: Array<string | null>,
|}
Referenced by:
SourceMapVLQMap source-map/src/types.js:22
type VLQMap = {
+sources: $ReadOnlyArray<string>,
+sourcesContent?: $ReadOnlyArray<string | null>,
+names: $ReadOnlyArray<string>,
+mappings: string,
+version?: number,
+file?: string,
+sourceRoot?: string,
}
Referenced by:
SourceMapSourceMapStringifyOptions source-map/src/types.js:33
type SourceMapStringifyOptions = {
file?: string,
sourceRoot?: string,
rootDir?: string,
inlineSources?: boolean,
fs?: {
readFile(path: string, encoding: string): Promise<string>,
...
},
format?: 'inline' | 'string' | 'object',
}