[#72] Add find-refs test coverage and fix its database/null-ref bugs (#79)

* Introduce second UnityProject

Rename UnityFileSystemTestData and introduce a LeadingEdge project for testing Content Directory builds and other newer features that require Unity >=6.6.
Add AGENTS.md explaining the relationship between these projects and the referenced files checked in to TestCommon

* Dump command tested with core numeric field types and other serialization primitives

* [#72] Fix find-refs database open and null-ref crashes, add test coverage

find-refs had no automated tests and two bugs that broke it against any
current analyze database:

- Issue #72: the database was opened with a hand-written connection string
  using the legacy System.Data.SQLite "Version=3" keyword, which
  Microsoft.Data.Sqlite rejects. Build the connection string with
  SqliteConnectionStringBuilder instead (matching SQLiteWriter). Fixed the
  same broken string in ExpectedDataGenerator.

- A ScriptableObject is a MonoBehaviour whose m_GameObject PPtr is 0, so its
  game_object column is a non-null id matching no row; the game_object/script
  subqueries then return NULL and GetString threw. Null-check those reads.

Also add a --stdout option to find-refs (mutually exclusive with -o), mirroring
dump, and update command-find-refs.md.

Add FindRefsTests covering both bugs, name/type lookup and disambiguation,
object-id lookup, --find-all, missing objects, empty refs table, the
--stdout/-o validator, and direct refs-table queries. Tests run against the
LeadingEdge AssetBundle reference build.

Author: SkowronskiAndrew

.gitignore

@@ -32,12 +32,13 @@
 
 .claude
 
-UnityFileSystemTestData/AssetBundles/
-UnityFileSystemTestData/**/*.csproj
-UnityFileSystemTestData/**/*.sln
-UnityFileSystemTestData/ProjectSettings/
-UnityFileSystemTestData/UserSettings/
-UnityFileSystemTestData/Packages/
+UnityProjects/**/*.csproj
+UnityProjects/**/*.sln
+UnityProjects/**/AssetBundles/
+UnityProjects/**/ProjectSettings/
+UnityProjects/**/UserSettings/
+UnityProjects/**/Packages/
+UnityProjects/**/Builds/
 *.db
 *.csv
 

AGENTS.md

@@ -174,4 +174,6 @@ To use a specific Unity version's library:
 
 * TestCommon/Data contains small reference files extracted from Unity builds (player and AssetBundles). These are used by the automated tests and also useful for manual testing.
 
-* UnityFileSystemTestData is a Unity project that generates test data for the test suites.
+* UnityProjects contains two Unity projects that generate test data for the test suites:
+  * `Baseline` - tracks stable, broadly-used Unity versions and is upgraded only when necessary. Most UnityDataTools users inspect output from these versions. Its `TypeIdRegistryGenerator` regenerates `UnityFileSystem/TypeIdRegistry.cs`.
+  * `LeadingEdge` - tracks the newest Unity version (currently the 6.6 beta) and is updated proactively so newer build features (Content Directory builds, serialized dictionaries, etc.) can be tested.

CLAUDE.md

@@ -1 +1 @@
-AGENTS.md
+@AGENTS.md

Documentation/command-find-refs.md

@@ -1,9 +1,9 @@
 # find-refs Command
 
-> ⚠️ **Experimental:** This command may not work as expected in all cases.
-
 The `find-refs` command traces reference chains leading to specific objects. Use it to understand why an asset was included (and potentially duplicated) in a build.
 
+It walks *up* the reference graph from the target object and **stops at the first asset it reaches**. The reported chains therefore end at the immediate containing asset, not at the ultimate root that transitively depends on the target. For example, if `RootAsset` references `LeafAsset` which references a texture, searching for the texture reports the chain ending at `LeafAsset`; to see that `RootAsset` pulls it in, search for `LeafAsset` instead.
+
 ## Quick Reference
 
 ```
@@ -16,10 +16,13 @@ UnityDataTool find-refs <database> [options]
 | `-i, --object-id <id>` | ID of object to analyze (from `id` column) | — |
 | `-n, --object-name <name>` | Name of objects to analyze | — |
 | `-t, --object-type <type>` | Type filter when using `-n` | — |
-| `-o, --output-file <file>` | Output filename | — |
+| `-o, --output-file <file>` | Output filename | `references.txt` |
+| `--stdout` | Write the reference chains to stdout instead of a file | `false` |
 | `-a, --find-all` | Find all chains instead of stopping at first | `false` |
 
 > **Note:** Either `--object-id` or `--object-name` must be provided.
+>
+> `--stdout` and `-o/--output-file` are mutually exclusive.
 
 ## Prerequisites
 
@@ -29,14 +32,19 @@ This command requires a database created by the [`analyze`](command-analyze.md)
 
 ## Examples
 
-Find references to an object by name and type:
+Find references to an object by name and type, printing directly to the console:
+```bash
+UnityDataTool find-refs my_database.db -n "MyTexture" -t "Texture2D" --stdout
+```
+
+Write the reference chains to a file instead:
 ```bash
 UnityDataTool find-refs my_database.db -n "MyTexture" -t "Texture2D" -o refs.txt
 ```
 
 Find references to a specific object by ID:
 ```bash
-UnityDataTool find-refs my_database.db -i 12345 -o refs.txt
+UnityDataTool find-refs my_database.db -i 12345 --stdout
 ```
 
 Find all duplicate references (useful for finding why an asset is duplicated):

README.md

@@ -25,7 +25,7 @@ The repository contains the following items:
 * UnityFileSystem: source code and binaries of a .NET class library exposing the functionalities or the
   UnityFileSystemApi native library.
 * UnityFileSystem.Tests: test suite for the UnityFileSystem library.
-* UnityFileSystemTestData: the Unity project used to generate the test data.
+* UnityProjects: Unity projects used to generate some of the test data.
 * TestCommon: a helper library used by the test projects.
 
 ## Downloads

ReferenceFinder/ReferenceFinderTool.cs

@@ -24,21 +24,14 @@ public class ReferenceFinderTool
     List<ReferenceTreeNode> m_Roots = new List<ReferenceTreeNode>();
     HashSet<(long, string)> m_ProcessedObjects = new HashSet<(long, string)>();
 
-    StreamWriter m_Writer;
+    TextWriter m_Writer;
 
-    public int FindReferences(string objectName, string objectType, string databasePath, string outputFile, bool findAll)
+    public int FindReferences(string objectName, string objectType, string databasePath, string outputFile, bool findAll, bool toStdout = false)
     {
         var objectIds = new List<long>();
-        SqliteConnection db;
-
-        try
-        {
-            db = new SqliteConnection($"Data Source={databasePath};Version=3;Foreign Keys=False;");
-            db.Open();
-        }
-        catch (Exception e)
+        using var db = OpenDatabase(databasePath);
+        if (db == null)
         {
-            Console.WriteLine($"Error opening database: {e.Message}");
             return 1;
         }
 
@@ -81,33 +74,50 @@ public int FindReferences(string objectName, string objectType, string databaseP
             return 1;
         }
 
-        return FindReferences(db, outputFile, objectIds, findAll);
+        return FindReferences(db, outputFile, objectIds, findAll, toStdout);
     }
 
-    public int FindReferences(long objectId, string databasePath, string outputFile, bool findAll)
+    public int FindReferences(long objectId, string databasePath, string outputFile, bool findAll, bool toStdout = false)
     {
         var objectIds = new List<long>();
-        SqliteConnection db;
+        using var db = OpenDatabase(databasePath);
+        if (db == null)
+        {
+            return 1;
+        }
 
+        objectIds.Add(objectId);
+
+        return FindReferences(db, outputFile, objectIds, findAll, toStdout);
+    }
+
+    // Opens the analyze database for reading. Uses SqliteConnectionStringBuilder (matching SQLiteWriter) rather than a
+    // hand-written connection string, which used a legacy System.Data.SQLite keyword that Microsoft.Data.Sqlite rejects.
+    static SqliteConnection OpenDatabase(string databasePath)
+    {
         try
         {
-            db = new SqliteConnection($"Data Source={databasePath};Version=3;Foreign Keys=False;");
+            var connectionString = new SqliteConnectionStringBuilder
+            {
+                DataSource = databasePath,
+                Mode = SqliteOpenMode.ReadOnly,
+                Pooling = false,
+                ForeignKeys = false,
+            }.ConnectionString;
+            var db = new SqliteConnection(connectionString);
             db.Open();
+            return db;
         }
         catch (Exception e)
         {
             Console.WriteLine($"Error opening database: {e.Message}");
-            return 1;
+            return null;
         }
-
-        objectIds.Add(objectId);
-
-        return FindReferences(db, outputFile, objectIds, findAll);
     }
 
-    int FindReferences(SqliteConnection db, string outputFile, IList<long> objectIds, bool findAll)
+    int FindReferences(SqliteConnection db, string outputFile, IList<long> objectIds, bool findAll, bool toStdout)
     {
-        m_Writer = new StreamWriter(outputFile);
+        m_Writer = toStdout ? Console.Out : new StreamWriter(outputFile);
 
         m_GetRefsCommand = db.CreateCommand();
         m_GetRefsCommand.CommandText = @"SELECT object, property_path, EXISTS (SELECT * FROM assets a WHERE a.object = r.object) FROM refs r WHERE referenced_object = @id";
@@ -181,7 +191,11 @@ FROM object_view o
             }
         }
 
-        m_Writer.Close();
+        // Don't close Console.Out when writing to stdout; just flush it.
+        if (toStdout)
+            m_Writer.Flush();
+        else
+            m_Writer.Close();
 
         return 0;
     }
@@ -196,10 +210,13 @@ void OutputReferenceNode(ReferenceTreeNode node, string propertyPath, int indent
         {
             reader.Read();
 
+            // game_object and script come from correlated subqueries that yield NULL when there is no matching row
+            // (e.g. a ScriptableObject is a MonoBehaviour whose m_GameObject PPtr is 0, or a MonoBehaviour with no
+            // m_Script reference), so both must be null-checked.
             var objectType = reader.GetString(0);
             var objectName = reader.GetString(1);
-            var gameObject = reader.GetString(2);
-            var script = reader.GetString(3);
+            var gameObject = reader.IsDBNull(2) ? "" : reader.GetString(2);
+            var script = reader.IsDBNull(3) ? "" : reader.GetString(3);
 
             if (propertyPath != "")
             {

TestCommon/Data/AssetBundles/AGENTS.md

@@ -0,0 +1,5 @@
+# AssetBundle test data
+
+AssetBundle build output used by the UnityDataTools test suites, kept in a subfolder per Unity version (`2019.4.0f1` through `2023.1.0a16`). Each holds the `assetbundle` and `scenes` bundles built from the project's assets; some versions also include additional fixtures used by specific tests.
+
+Generated by the `UnityProjects/Baseline` project - run **Tools > Generate AssetBundles** there (built for StandaloneOSX and copied here under the Unity version). See that project's `AGENTS.md` for the assets that go into the bundles.

TestCommon/Data/AssetBundles/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

TestCommon/Data/LeadingEdgeBuilds/AGENTS.md

@@ -0,0 +1,12 @@
+# LeadingEdge reference builds
+
+The checked-in build output of the `UnityProjects/LeadingEdge` project, for use in automated and ad hoc tests. See that project's `AGENTS.md` for the test scenarios and how the assets are set up.
+
+The LeadingEdge build scripts regenerate this folder directly, so to update it, rebuild in that project and check in the results.
+
+## Layout
+
+* `AssetBundles/` - the AssetBundle build: one bundle per asset (named after the asset) plus the `AssetBundles` manifest bundle.
+* `ContentDirectory/` - the Content Directory build: content (`.cf`) files, `.resource` files and the build manifest.
+* `BuildReport-AssetBundles/LastBuild.buildreport` - the AssetBundle build report.
+* `BuildReport-ContentDirectory/` - the Content Directory build report folder, including `ContentLayout.json`.