{"id":26,"date":"2022-05-01T20:11:55","date_gmt":"2022-05-01T20:11:55","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/?page_id=26"},"modified":"2022-12-20T21:23:59","modified_gmt":"2022-12-20T21:23:59","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/","title":{"rendered":"Method"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Method Overview. <\/strong>Our search system consists of a pre-caching stage (a, b) and an inference stage (c). Given a collection of models \ud835\udf03 \u223c unif{\ud835\udf031, \ud835\udf032, . . . , \ud835\udf03\ud835\udc41 }, (a) we first generate 50K samples for each model \ud835\udf03\ud835\udc5b. (b) We then encode the images into image features and compute the 1st and 2nd order feature statistics for each model. The statistics are cached in our system for efficiency. (c) At inference time, we support queries of different modalities (text, image, or sketch). We encode the query into a feature vector and assess the similarity between the query feature and each model\u2019s statistics. The models with the best similarity measures are retrieved.<\/em><\/p>\n\n\n\n

Model retrieval is a challenging task: even the simplified question of whether a specific image can be produced by a single model can be computationally difficult. Unfortunately, many deep generative models do not offer an efficient or exact way to estimate density, nor do they natively support assessing cross-modal similarity (e.g., text and image). A naive Monte Carlo approach can compare the input query to thousands or even millions of samples from each generative model, and identify the model whose samples most often match the input query. However, such an approach can make model search too slow. <\/p>\n\n\n\n

We first present a general probabilistic formulation of the model search problem and present a Monte Carlo baseline. To reduce the search time and storage, we \u201ccompress\u201d the model\u2019s distribution into precomputed 1st and 2nd order moments of the deep feature embeddings of the original samples. We then derive closed-form solutions for model retrieval given an input image, text, sketch, or model query. Our final formula can be evaluated in real-time.<\/p>\n\n\n\n

Probabilistic Retrieval for Generative Models<\/strong><\/h2>\n\n\n\n

Our goal is to quantify the likelihood of each model \u03b8n given the user query q, by evaluating the conditional probability p(\u03b8|q). The model with the highest conditional probability is retrieved:<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

Since we assume the model collection is uniformly distributed, by Bayes\u2019 rule it suffices to find the likelihood of the query q given each model \u03b8 instead. There are two scenarios to infer the conditional probability p(q|\u03b8):<\/p>\n\n\n\n

(1) When q shares the same modality with \u03b8 (e.g., searching image generative models with image queries), we directly reduce the problem to estimating the generative model\u2019s density. <\/p>\n\n\n\n

(2) When q has a different modality to \u03b8 (e.g., searching image generative models with text queries), we take account of cross-modal similarity to estimate p(q|\u03b8). We discuss the two cases in the following.<\/p>\n\n\n\n

Image-based Model Retrieval<\/strong><\/h2>\n\n\n\n

Given an image query q, we directly estimate the likelihood of the query p(q|\u03b8) from each model. In other words, the best-matched model is the one most likely to generate the query image. Since density is intractable or inaccurate for many generative models, we approximate each model by a Gaussian distribution of image features. We sample images from each model, denoted by x. We obtain the image features z = \u03c8im<\/sub>(x), where \u03c8im<\/sub> is the feature extractor. Now we express p(q|\u03b8) in terms of image features z.<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

where the query image feature is denoted by zq<\/sub> = \u03c8im<\/sub>(q), and each model \u03b8 is approximated by p(z|\u03b8) \u223c N(\u00b5n<\/sub>, \u03a3n<\/sub>). We refer to this method as Gaussian Density.<\/strong><\/em> <\/p>\n\n\n\n

We can use the same method for sketch-based model retrieval<\/strong> if the embedding network \u03c8im<\/sub> also works for human sketches. In our experiment, we find that CLIP can produce similar feature embeddings for similar images and sketches. CLIP outperforms other pre-trained networks (e.g., DINO) by a large margin.<\/p>\n\n\n\n

Text-based Model Retrieval<\/strong><\/h2>\n\n\n\n

Given a text query q and a generative model p(x|\u03b8) capturing a distribution of images x, we want to estimate the conditional probability p(q|\u03b8).<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

Here we assume conditional independence between query q and model \u03b8 given image x, so p(q|x, \u03b8) = p(q|x). We apply Bayes\u2019 rule to get the final expression. The text query q may correspond to multiple possible image matches p(x|q), and we estimate the term p(x|q)\/p(x) using cross-modal similarities. In fact, this expression is proportional to the score function f in contrastive learning (e.g., InfoNCE), where f(x, q) is proportional to p(x|q)\/p(x). Since CLIP is trained on a text-image retrieval task with the InfoNCE loss, we can directly apply the pre-trained CLIP model to simplify the above equation. <\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

We recall that CLIP consists of an image encoder \u03c6im<\/sub> and a text encoder \u03c6txt<\/sub>, and it is trained with a score function based on cosine similarity:<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

where hx<\/sub> = \u03c6im<\/sub>(x) and hq<\/sub> = \u03c6txt(q) are the image and text features from CLIP, respectively. h\u02dcx = hx<\/sub>\/||hx<\/sub>|| and h\u02dcq<\/sub> = hq<\/sub>||hq<\/sub>|| are the normalized features. Hence, Equation 5 can be written precisely as:<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

Now we have a tractable Monte Carlo estimate of the integral. We sample images from each model and average the score function of each image sample x and the text query q. We refer to this method as Monte Carlo<\/strong>. However, directly applying Monte Carlo estimation is inefficient in practice, since we need lots of samples to yield a robust estimate. To speed up computation, we provide two ways to approximate the above equation. First, we find that a point estimate at the first moment of p(hx<\/sub>|\u03b8) works well. We directly estimate the cosine distance between the mean image features and the query feature. Since the exponential and temperature mapping is monotonically increasing, the matching function becomes:<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

We refer to this method as 1st Moment<\/strong>. We can also approximate p(hx<\/sub>|\u03b8) using both the first and the second moment to get the following expression.<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

We refer to this method as 1st + 2nd Moment<\/strong>. Empirically, the performance is similar between approximation to the first or second moment. We provide more analysis in the paper.<\/a><\/p>\n\n\n\n

Extensions<\/strong><\/h2>\n\n\n\n

Multi-modal query.<\/strong> We can further extend our model search to handle multiple queries from different modalities. To achieve this, we use a Product-of-Experts formulation [Hinton 2002; Huang et al. 2021] wherein the final likelihood of a model, given a multimodal query (e.g., text-image pair), is modeled as a product of likelihoods given individual queries followed by a renormalization. <\/p>\n\n\n\n

Finding similar models.<\/strong> Once a model is found, we enable navigation to similar models. To compute the similarity between models, we use the Fr\u00e9chet Distance [Dowson and Landau 1982] between the models\u2019 feature distributions. Following prior work [Heusel et al. 2017a; Kynk\u00e4\u00e4nniemi et al. 2022], we approximate a model\u2019s distribution by fitting a multivariate Gaussian in an image feature space. Then the Fr\u00e9chet Distance can be computed directly from the Gaussian parameters. For each model, we pre-compute a list of similar models based on the smallest pairwise distances.<\/p>\n\n\n\n

User Interface<\/strong><\/h2>\n\n\n\n
\"\"<\/figure><\/div>\n\n\n\n

The user interface of model search.<\/strong> The user can enter a text query
and\/or upload an image in the search bar to retrieve generative models
that best match the query. Here we show the top retrievals for the text query
\u201canimated faces,\u201d which shows StyleGAN-NADA models [Gal et al. 2022]
trained on animated characters.<\/em><\/p>\n\n\n\n

We create a web-based UI for our search algorithm. The UI supports searching and sampling from deep generative models in real time. The user can enter a text prompt, upload an image\/sketch, or provide both text and an image\/sketch. The interface displays the models that match most closely with the query. Clicking a model takes the user to a new page where they can sample new images from the model. The website employs a backend GPU server to enable real-time model search and image synthesis capabilities.<\/p>\n\n\n\n

References<\/strong><\/p>\n\n\n\n

OpenAI CLIP<\/a><\/p>\n\n\n\n

Representation Learning with Contrastive Predictive Coding<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Method Overview. Our search system consists of a pre-caching stage (a, b) and an inference stage (c). Given a collection of models \ud835\udf03 \u223c unif{\ud835\udf031, \ud835\udf032, . . . , \ud835\udf03\ud835\udc41 }, (a) we first generate 50K samples for each model \ud835\udf03\ud835\udc5b. (b) We then encode the images into image features and compute the 1st … <\/p>\n

Continue reading “Method”<\/span><\/a><\/p>\n","protected":false},"author":135,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-26","page","type-page","status-publish","hentry"],"yoast_head":"\nMethod - One Million Generative Models<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - One Million Generative Models\" \/>\n<meta property=\"og:description\" content=\"Method Overview. Our search system consists of a pre-caching stage (a, b) and an inference stage (c). Given a collection of models \ud835\udf03 \u223c unif{\ud835\udf031, \ud835\udf032, . . . , \ud835\udf03\ud835\udc41 }, (a) we first generate 50K samples for each model \ud835\udf03\ud835\udc5b. (b) We then encode the images into image features and compute the 1st … Continue reading "Method"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/\" \/>\n<meta property=\"og:site_name\" content=\"One Million Generative Models\" \/>\n<meta property=\"article:modified_time\" content=\"2022-12-20T21:23:59+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-content\/uploads\/sites\/63\/2022\/09\/image-6-1024x390.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/method\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/method\\\/\",\"name\":\"Method - One Million Generative Models\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/method\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/method\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/wp-content\\\/uploads\\\/sites\\\/63\\\/2022\\\/09\\\/image-6-1024x390.png\",\"datePublished\":\"2022-05-01T20:11:55+00:00\",\"dateModified\":\"2022-12-20T21:23:59+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/method\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/method\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/method\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/wp-content\\\/uploads\\\/sites\\\/63\\\/2022\\\/09\\\/image-6.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/wp-content\\\/uploads\\\/sites\\\/63\\\/2022\\\/09\\\/image-6.png\",\"width\":1648,\"height\":628},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/method\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/\",\"name\":\"One Million Generative Models\",\"description\":\"Students: Daohan "Fred" Lu (CMU), Rohan Agarwal (CMU) | Collaborators: Nupur Kumari (CMU), Sheng-Yu Wang (CMU) | Advisors: Jun-Yan Zhu (CMU), David Bau (Northeastern University)\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team8\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - One Million Generative Models","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/","og_locale":"en_US","og_type":"article","og_title":"Method - One Million Generative Models","og_description":"Method Overview. Our search system consists of a pre-caching stage (a, b) and an inference stage (c). Given a collection of models \ud835\udf03 \u223c unif{\ud835\udf031, \ud835\udf032, . . . , \ud835\udf03\ud835\udc41 }, (a) we first generate 50K samples for each model \ud835\udf03\ud835\udc5b. (b) We then encode the images into image features and compute the 1st … Continue reading \"Method\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/","og_site_name":"One Million Generative Models","article_modified_time":"2022-12-20T21:23:59+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-content\/uploads\/sites\/63\/2022\/09\/image-6-1024x390.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/","name":"Method - One Million Generative Models","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-content\/uploads\/sites\/63\/2022\/09\/image-6-1024x390.png","datePublished":"2022-05-01T20:11:55+00:00","dateModified":"2022-12-20T21:23:59+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-content\/uploads\/sites\/63\/2022\/09\/image-6.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-content\/uploads\/sites\/63\/2022\/09\/image-6.png","width":1648,"height":628},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/method\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/","name":"One Million Generative Models","description":"Students: Daohan "Fred" Lu (CMU), Rohan Agarwal (CMU) | Collaborators: Nupur Kumari (CMU), Sheng-Yu Wang (CMU) | Advisors: Jun-Yan Zhu (CMU), David Bau (Northeastern University)","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-json\/wp\/v2\/pages\/26","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-json\/wp\/v2\/users\/135"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-json\/wp\/v2\/comments?post=26"}],"version-history":[{"count":19,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-json\/wp\/v2\/pages\/26\/revisions"}],"predecessor-version":[{"id":287,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-json\/wp\/v2\/pages\/26\/revisions\/287"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team8\/wp-json\/wp\/v2\/media?parent=26"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}