ReCollection Iṣesí Iṣẹ Ọnà Onífaramọ láti ọwọ́ Weidi Zhang and Jieliang Luo

ReCollection Iṣesí Iṣẹ Ọnà Onífaramọ láti ọwọ́ Weidi Zhang and Jieliang Luo jẹ́ aṣẹ́gun olókìkí púpọ̀ ti 2024 Àmì-ẹ̀yẹ Idẹ A' Design Award nínú Ẹ̀ka Ìdíje Àpẹẹrẹ Èrò àti Àbá.

IdẹÈrò· 2024

ReCollection Iṣesí Iṣẹ Ọnà Onífaramọ

látọ̀wọ́ Weidi Zhang and Jieliang Luo

Aṣẹ́gun Àmì-ẹ̀yẹ Idẹ A' Design Award nínú Ẹ̀ka Èrò

84,717

Aṣẹ́gun Àmì Ẹ̀yẹ Tí a Jẹ́rìísí

Jẹ́rìísí Ipò Àmì Ẹ̀yẹ fún #157648

Ṣàbẹ̀wò Ojú Òpó Wo ní A' Design Award Ìwé Ìròyìn Àmì Ẹ̀yẹ

Àwọ̀

Àpẹrẹ Náà

Nípa ReCollection Iṣesí Iṣẹ Ọnà Onífaramọ

ReCollection jẹ́ ohun ìṣàwárí iṣẹ́ ọnà onífaramọ tí ó ń ṣẹ̀dá àwòrán láti inú ìrántí ẹnìkọ̀ọ̀kan nípasẹ̀ ọ̀rọ̀ sísọ. Àwọn tí ó ń kópa a máa fi ohùn kẹ́lẹ́kẹ́lẹ́ sọ àjákù ìrántí kan sínú microphone. Ẹ̀rọ ìmọ̀ ẹ̀rọ ayédèrú (artificial intelligence) a máa yí ọ̀rọ̀ sísọ náà padà sí àkọsílẹ̀ ọ̀rọ̀, kí ó parí gbólóhùn náà gẹ́gẹ́ bí ìtàn kékeré kan, kí ó sì ṣẹ̀dá àwòrán láti inú ìtàn náà. Àwọn àwòrán náà ni a óò wá ṣe àyẹ̀wò lẹ́sẹẹsẹ nípasẹ̀ algorithm láti dá àwòrán tí ń yí padà ní àkókò gidi. Ohun ìṣàwárí yìí ń wádìí àjọṣe tí ó wà láàrin èdè, ìrántí, àti agbára ìrò-inú, ó sì tún jẹ́ àpẹẹrẹ àṣesiwaju (prototype) fún iṣẹ́ ọjọ́ iwájú pẹ̀lú àwọn ẹni tí àìsàn ìgbàgbé (dementia) ń kàn.

Ìdánimọ̀ Náà

Nípa Àmì-ẹ̀yẹ Idẹ A' Design Award

ReCollection Iṣesí Iṣẹ Ọnà Onífaramọ jẹ́ olùborí Àmì-ẹ̀yẹ Idẹ A' Design Award, ìdánimọ̀ olókìkí gan-an tí a fún àwọn àwòṣe tó tayọ tí wọ́n yàtọ̀ nípasẹ̀ ìṣẹ̀dá, àrékérekè àti ọgbọ́n inú. Àwọn olùborí Idẹ jẹ́ àwọn ohun èlò tó ní ìjáfáfá gan-an tí wọ́n sì ní ìṣẹ̀dá tó yani lẹ́nu tí wọ́n ń ṣe ẹ̀rí ìrírí gbígbòòrò, ìrònú àti ọgbọ́n àwọn olùṣàwòṣe wọn. A retí pé ReCollection Iṣesí Iṣẹ Ọnà Onífaramọ yóò jẹ́ ohun tí a kà sí gidigidi, tí a gbóríyìn fún tí a sì nífẹ̀ẹ́, tó jẹ́ ti ọ̀jáfáfá lọ́nà tó ṣe pàtàkì tó sì ní ìmúṣẹ̀dá tuntun. Àwòṣe tó borí Àmì-ẹ̀yẹ Idẹ A' Design Award yìí ń kó àwọn ọ̀nà tó dára jùlọ nínú iṣẹ́ ọnà, sáyẹ́ǹsì, àwòṣe àti ìmọ̀-ẹ̀rọ jọ, ó sì tayọ nítorí àfiyèsí rẹ̀ sí àwọn àlàyé kékeré. ReCollection Iṣesí Iṣẹ Ọnà Onífaramọ ń tẹ́ àwọn ohun tí a nílò lọ́rùn lọ́nà àgbàyanu, ó ń fọwọ́ kan àwọn ìfẹ́ inú, ó ń gbà ọkàn ó sì ń fa ìfẹ́ láti mọ̀ mọ́ra.

Lẹ́yìn Àpẹrẹ Náà

Àkójọ Ìròyìn

Àwọn ẹ̀tọ́, àwọn ọmọ ẹgbẹ́ àti àlàyé ìmọ̀ ẹ̀rọ tí olùṣàpẹrẹ pèsè.

Àwọn Ohun Àkànṣe: ReCollection is an AI art experience that synthesizes personal memories based on users’ language input, blurring the boundaries between remembrance and imagination through AI system design and experimental visualization. Beyond its potential as a future therapeutic prototype for dementia groups, this work provides new possibilities for collective memories and cultural reproduction.
Ìwọ̀n àti Àpò: Space Dimensions: While adaptable to different room sizes, we suggest a space dimension of 7m x 7m x 4m (L x W x H) to achieve the desired projection size and quality.

ReCollection was displayed as an interactive art installation incorporating a wall projection or an LED screen, and a pedestal positioned to face the projection. This pedestal was designed to hold a microphone and a USB button for interactivity. Our exhibition space was intentionally designed in a dark environment, lit by the projection light and a dimmed spotlight situated atop the pedestal.

Technical Properties:

1. Audio Setup: A speaker tailored for the audio component.

2. Main Pedestal: This will house both the USB microphone and a USB button.

3. Instruction Pedestal: A separate pedestal dedicated solely for the placement of instructions.

4. A USB Microphone

5. A PC
6. A 4K capable projector.
Àwọn Àkọlé: generative design, data visualization, interactive AI art
Ìmọ̀-ẹ̀rọ Ìṣelọ́pọ̀: ReCollection integrates AI systems and experimental visualization into the design of an interactive art experience. By employing AI techniques like speech recognition, text auto-completion, and text-to-image models, it translates voice inputs of fragmented language into moving computer graphics in real-time. In the art installation, ReCollection invites participants to share their personal memories in one sentence. The AI system then transforms speech to text and automatically fills in details as a completed narrative, generating new visual memories.
Ìpèníjà Àpèsè: In ReCollection, AI is engaged in the decision-making process in imaginative acts. Presently, AI's role in dementia research spans from early detection and progression mapping of the disease to serving as companion for individuals with dementia. However, the utilization of AI in dementia research often remains confined to caregiving roles, rarely exploring its potential in more symbiotic, imaginative collaborations. Drawing inspiration from the development of dementia research and its critical turn, ReCollection investigates connections among language, memory, and imagination through integrating AI as a non-human narrative agency in collaborative worldmaking, investigating the artistic expression of memory and narratives.

We have developed an interactive AI-powered story-generation system, comprising three major components that synergize to create a comprehensive narrative experience, each contributing to a seamless storytelling process.

The system begins with Whisper an advanced speech-to-text model developed by OpenAI, capable of transcribing spoken words into written text. Significantly, Whisper supports 98 languages. For the second component, we employ GPT- 4 for text completion. Drawing inspiration from documentaries on Alzheimer's, such as A Marriage to Remember. we've defined two distinct agents in the prompts. The first agent controls the content, ensuring the narrative is coherent and contextually relevant, while the second agent influences the visual style, akin to how a camera lens or shooting angle might affect the visualization in a film. This dual-agent approach effectively creates a storyboard in written form, adding a layer of depth and sophistication to the generative narrative.

Lastly, our system integrates Stable Diffusion APIs provided by Replicate. Stable Diffusion is a text-to-image model, to convert the written narrative into vivid visual representations. This not only enriches the storytelling with illustrative content but also ensures that the visual output is as engaging and immersive as the written story. Collectively, these components synergize to create a dynamic narrative experience for creative expression. We notice that our system can culturally adapt the visual output to match the input language. This allows for a nuanced and culturally sensitive storytelling experience, enhancing the relevance and immersion of the narrative. Leveraging the collective strengths of each AI model, ReCollection produces a personalized and engaging user experience that pushes the boundaries of traditional storytelling.
Ìwádìí: This artwork was born of witnessing my grandmother's memory regression due to dementia, where her cherished stories dissolved into fragmented words. Dr. Mary Steedly once described memories as a "densely layered, sometimes conflictual negotiation with the passage of time", and in 2022, over 50 million people faced this painful reality of memory loss due to Alzheimer's and related dementias. Yet, amidst this poignant backdrop, the emergence of text-to-image AI systems in 2022 offered a glimmer of new perspective, as they harnessed the power of language to imagine and reassemble fragmented memories, possibly to weave what time and disease had stolen.

When we coexist with machines, will we accumulate synthetic recollections of collective symbiotic imagination?

Is language capable of re-weaving and synthesizing memories?

How does our collective memory inspire new visual forms and alternative narratives?

Recollection is an assemblage of intimate human-machine artifacts that emphasizes the contributions from three sides: artists, machines, and participants. This customized AI application facilitates multiple AI techniques, like speech recognition, text auto-completion, and text-to-image, to convert language input into image sequences of new memories. As an interactive experience, participants will whisper their personal memories with fragmented sentences, and our system will automatically fill in details, creating new touching visual memories.

We developed our customized AI system by fine-tuning a pre-trained transformer-based AI model to learn the documentaries of Alzheimer patients’ visual memories and their descriptions. The system imagines new memories of "love" and "loss" by interpreting real-time narratives from participants in the installation. Our system emerges as a vibrant and inclusive conversation starter, transcending boundaries with support for over 89 different languages, embracing the diverse cultural artifacts.

We did not employ the generative visual output from our AI system as our final delivery in the installation. Instead, we draw inspiration from Monotype, a printmaking technique with a history that dates back to the 1640s, which produces unique visualizations on paper through printing and reprinting. Therefore, we program our system to automatically transform the machine-generated visuals algorithmically into an evolving real-time visualization with a unique aesthetic through image processing and experimental data visualization.

By providing a conceptual framework for non-linear narratives, which constitute symbiotic imaginations, and future scenarios of memories, culture production, and reproductions. It may inspire the cure for memory regression by providing a future scenario, a thought experiment, and an intimate recollection of symbiosis between beings and apparatus. It raises people's awareness of future memory preservation and their empathy for the dementia community through a personalized aesthetic experience. It offers an artistic approach and future prototype for cultural heritage reproduction and re-imagination and explores the tensions that exist in the co-relations between visual representations, language, and narratives.
Ìmísí: ReCollection was born of witnessing my grandmother's memory regression due to dementia, where her cherished stories dissolved into fragmented words. Since 2020, over 50 million people have faced this painful reality of memory loss due to Alzheimer's and related dementias. Yet, amidst this poignant backdrop, the emergence of text-to-image AI systems in 2022 offered a glimmer of new perspective, as they harnessed the power of language to imagine and reassemble fragmented memories.
Àkókò Iṣẹ́ Àkànṣe: Duration varied (Interactive work)
Location: this project has been presented in Los Angeles Convention Center (United States), Signal Immersive Art Gallery (Canada), Media and Immersive eXperience Center (United States), and will be presented at Cinema Mystica (Hungary)
Ìlànà Ìṣiṣẹ́: In the art installation, a participant will whisper fragmented memories into the microphone. The AI system will automatically fill in the details of the spoken words to complete the text with a narrative using the GPT-4, a large language model. The completed narrative will be sent to Stable Diffusion to generate synthetic images representing the memories based on the machine’s interpretation. The images output by machines is further developed and visualized algorithmically as an evolving interactive experience.
Àwọn Ọmọ Ẹgbẹ́: Weidi Zhang
Jieliang Luo
Ìfọwọ́sí Àwòrán: credit to Weidi Zhang and Jieliang Luo