[ React Native ] 키보드로 조작가능한 앱 만들기

지금 다니는 회사에서 Barrier Free 기능을 테블릿오더에 탑재하기로 하였다.

이는 장애우분들도 테블릿오더로 사용하기위함이었다.

해당 기능에는 시각장애우모드가 있기때문에, 모든 컴포넌트들은 키보드로 사용가능하도록(방향키 / 선택 등) 처음 구성하였다.

 

기본적인 개념은 모든 기능은 스크롤이 없이, 터치만으로 사용이 가능하여야하고, 해당 터치모드는 시각장애우들이 이어폰을 꽂았을때, 키보드로 조작이 가능하기로 하였다.

 

총 작업기간은 주말출근 등 빡시게 하여 한달만에 완성하였다. 물론 QA테스트등을 위하여 실 워킹데이는 22일정도 된다.

해당 작업을 받았을때,

일단 터치만으로 사용가능한 컴포넌트들을 구성하고, 추후에 ref 값으로 키보드 포커싱을 이동하는 개념으로 잡았다.

 

터치만으로 작업시에, FlatList 들을 pagination 을 통해 , index 를 관리하고, 터치로, 해당 스크롤들 조작하는 방법으로 했다.

 

물론 해당 기능은 크게 어렵지않았는데,

 

시각장애우분들을 위한 키패드모드를 하면서 ref에 대한 공부가 많이 된것같다.

 

기본적으로

 

import React, {
  ReactNode,
  useEffect,
  useRef,
  forwardRef,
  useImperativeHandle,
} from 'react';
import {
  Animated,
  Pressable,
  PressableProps,
  StyleSheet,
  Text,
  ViewStyle,
} from 'react-native';
import { TSoundType, usePlayPollyTTS } from '@/utils/polly/playPollyTTS';

const AnimatedPressable = Animated.createAnimatedComponent(Pressable);
const AnimatedText = Animated.createAnimatedComponent(Text);

interface BFTouchableProps extends PressableProps {
  children: ReactNode;
  style?: ViewStyle | ViewStyle[];
  type?: TSoundType;
  activeBackgroundColor?: string;
  activeTextColor?: string;
  activeBorderColor?: string;
  sound?: string;
  soundKey?: string;
}

const BFTouchable = forwardRef<any, BFTouchableProps>(
  (
    {
      children,
      style = {},
      onPress,
      onLongPress,
      type,
      activeBackgroundColor,
      activeTextColor,
      activeBorderColor,
      ...props
    },
    ref,
  ) => {
    const localRef = useRef<any>(null); // 내부 ref
    const propsRef = useRef(props);
    React.useEffect(() => {
      propsRef.current = { ...props, onPress };
    }, [onPress, props]);
    useImperativeHandle(ref, () => ({
      ...localRef.current,
      focus: () => {
        if (props?.sound) {
          playPollyTTS({ text: props?.sound, key: props?.soundKey });
        } else {
          playPollyTTS({ text: '항목이 선택되었습니다', key: 'focused' });
        }
      },
      press: () => {
        if (propsRef.current?.onPress) {
          propsRef.current.onPress({ nativeEvent: {} } as any);
        }
      },
    }));

    const intervalRef = useRef<NodeJS.Timeout | null>(null);
    const animatedValue = useRef(new Animated.Value(0)).current;
    const flattenedStyle = StyleSheet.flatten(style);
    const { playPollyTTS } = usePlayPollyTTS();

    const defaultBackgroundColor =
      flattenedStyle?.backgroundColor || 'transparent';
    const defaultBorderColor = flattenedStyle?.borderColor || 'transparent';

    let extractedTextColor: string | undefined;
    React.Children.forEach(children, (child) => {
      if (
        React.isValidElement(child) &&
        child.type === Text &&
        child.props.style
      ) {
        const textStyle = StyleSheet.flatten(child.props.style);
        if (textStyle?.color) {
          extractedTextColor = textStyle.color;
        }
      }
    });

    const defaultTextColor = extractedTextColor || 'black';

    const backgroundColor = animatedValue.interpolate({
      inputRange: [0, 1],
      outputRange: [
        defaultBackgroundColor,
        activeBackgroundColor || defaultBackgroundColor,
      ],
    });

    const borderColor = animatedValue.interpolate({
      inputRange: [0, 1],
      outputRange: [
        defaultBorderColor,
        activeBorderColor || defaultBorderColor,
      ],
    });

    const textColor = animatedValue.interpolate({
      inputRange: [0, 1],
      outputRange: [defaultTextColor, activeTextColor || defaultTextColor],
    });

    const handlePressIn = () => {
      if (!activeBackgroundColor && !activeTextColor && !activeBorderColor)
        return;
      Animated.timing(animatedValue, {
        toValue: 1,
        duration: 0,
        useNativeDriver: false,
      }).start(() => {
        setTimeout(() => {
          Animated.timing(animatedValue, {
            toValue: 0,
            duration: 0,
            useNativeDriver: false,
          }).start();
        }, 300);
      });
    };

    const handleLongPressStart = (event: any) => {
      if (onPress) onPress(event);
      playPollyTTS({ text: 'BF', key: 'click' });

      intervalRef.current = setInterval(() => {
        if (onPress) onPress(event);
        playPollyTTS({ text: 'BF', key: 'click' });
      }, 300);
    };

    const handleLongPressEnd = () => {
      if (intervalRef.current) {
        clearInterval(intervalRef.current);
        intervalRef.current = null;
      }
    };

    useEffect(() => {
      return () => {
        if (intervalRef.current) {
          clearInterval(intervalRef.current);
          intervalRef.current = null;
        }
      };
    }, []);

    const renderChildren = () => {
      return React.Children.map(children, (child) => {
        if (React.isValidElement(child) && child.type === Text) {
          return (
            <AnimatedText
              {...child.props}
              style={[child.props.style, { color: textColor }]}
            >
              {child.props.children}
            </AnimatedText>
          );
        }
        return child;
      });
    };

    return (
      <AnimatedPressable
        ref={localRef}
        android_disableSound={true}
        style={[style, { backgroundColor, borderColor }]}
        onPress={(event) => {
          handlePressIn();
          playPollyTTS({ text: 'BF', key: 'click' });
          if (onPress) onPress(event);
        }}
        onPressIn={handlePressIn}
        onPressOut={handleLongPressEnd}
        onPressCancel={handleLongPressEnd}
        onLongPress={onLongPress ? handleLongPressStart : undefined}
        {...props}
      >
        {renderChildren()}
      </AnimatedPressable>
    );
  },
);

BFTouchable.displayName = 'BFTouchable';

export default BFTouchable;

 

모든 버튼에서 사용할 컴포넌트를 만들었다.

 

함수형 컴포넌트들의 ref 값을 참조하기위해 useImperativeHandle 훅을 사용하여 ref 값을 설정해주었고,

 

키보드를 통해 focus 가 될때마다 음성인식을 해주었다.

 

onPress 로 받은 함수를 실행할때도, ref 값의 할당은 UI 가 다 그려지고나서 하기때문에 안에서 ref 값을 한번 더 사용해주었다.

 

import useModal from '@/utils/hooks/useModal';
import { usePlayPollyTTS } from '@/utils/polly/playPollyTTS';
import { useIsFocused } from '@react-navigation/native';
import React from 'react';
import KeyEvent from 'react-native-keyevent';
import { useBFContext } from './BFContext';

export const BFKeyType = {
  volumeUp: 137,
  volumeDown: 136,
  mode: 132,
  arrowRight: 22,
  arrowLeft: 21,
  center: 112,
  arrowUp: 19,
  arrowDown: 20,
  home: 122,
  info: 131,
  back: 111,
  jackIn: 138,
  jackOut: 139,
};

type BFKeyCode = (typeof BFKeyType)[keyof typeof BFKeyType];

type TUseKeypad = {
  narration?: { key?: string; text: string };
  onArrowUp?: () => void;
  onBack?: () => void;
  focusableRefs?: React.MutableRefObject<React.Component<any, any, any>[]>;
  onInfo?: () => void;
  onHome?: () => void;
  visible?: boolean;
  currentIndexRef: React.MutableRefObject<number>;
  onSelect?: () => void;
  setMilliseconds?: React.Dispatch<React.SetStateAction<number>>;
  onReset?: () => void;
  setModalSeconds?: React.Dispatch<React.SetStateAction<number>>;
};

export const useKeypad = ({
  narration,
  onArrowUp,
  onBack,
  focusableRefs,
  onInfo,
  onHome,
  visible,
  currentIndexRef,
  onSelect,
  setMilliseconds,
  onReset,
  setModalSeconds,
}: TUseKeypad) => {
  const { replayLastAudio, playPollyTTS, stopCurrentAudio } = usePlayPollyTTS();
  const { currentVolume, currentSpeed, isSettingVoice, isTimeDone } =
    useBFContext();
  const modal = useModal();
  const isFocused = useIsFocused();

  const settingSpeed = React.useCallback(
    (keyCode: BFKeyCode, speed: number) => {
      if (speed === 1.5) {
        if (keyCode === BFKeyType.arrowRight) {
          playPollyTTS({ text: 'stuck', key: 'stuck' });
        } else if (keyCode === BFKeyType.arrowLeft) {
          currentSpeed.current = 1;
          playPollyTTS({
            text: '현재 보통 속도로 말하고 있어요. 좌우 방향키를 이용해 속도를 조절해보세요',
            key: 'settingSpeed_1',
          });
        }
      } else if (speed === 0.8) {
        if (keyCode === BFKeyType.arrowRight) {
          currentSpeed.current = 1;
          playPollyTTS({
            text: '현재 보통 속도로 말하고 있어요. 좌우 방향키를 이용해 속도를 조절해보세요',
            key: 'settingSpeed_1',
          });
        } else if (keyCode === BFKeyType.arrowLeft) {
          playPollyTTS({ text: 'stuck', key: 'stuck' });
        }
      } else if (speed === 1) {
        if (keyCode === BFKeyType.arrowRight) {
          currentSpeed.current = 1.5;
          playPollyTTS({
            text: '현재 더 빠른속도로 말하고있어요',
            key: 'settingSpeed_2',
          });
        } else if (keyCode === BFKeyType.arrowLeft) {
          currentSpeed.current = 0.8;
          playPollyTTS({
            text: '현재 더 느린 속도로 말하고 있어요',
            key: 'settingSpeed_3',
          });
        }
      }
    },
    [],
  );

  const settingVolume = React.useCallback(
    (keyCode: BFKeyCode, volume: number) => {
      const speak = (text: string, key: string) => playPollyTTS({ text, key });

      if (volume === 0.99) {
        currentVolume.current = 0.33;
        keyCode === BFKeyType.volumeUp
          ? speak('현재 더 작게 말하고있어요', 'settinvVolume_2')
          : keyCode === BFKeyType.volumeDown &&
            ((currentVolume.current = 0.66),
            speak('현재 보통 크기로 말하고있어요', 'settinvVolume_1'));
      } else if (volume === 0.33) {
        keyCode === BFKeyType.volumeUp &&
          ((currentVolume.current = 0.66),
          speak('현재 보통 크기로 말하고있어요', 'settinvVolume_1'));
        keyCode === BFKeyType.volumeDown &&
          ((currentVolume.current = 0.99),
          speak('현재 더 크게 말하고 있어요', 'settinvVolume_3'));
      } else if (volume === 0.66) {
        keyCode === BFKeyType.volumeUp &&
          ((currentVolume.current = 0.99),
          speak('현재 더 크게 말하고 있어요', 'settinvVolume_3'));
        keyCode === BFKeyType.volumeDown &&
          ((currentVolume.current = 0.33),
          speak('현재 더 작게 말하고있어요', 'settinvVolume_2'));
      }
    },
    [],
  );

  const settingVoice = React.useCallback(
    async (keyCode: BFKeyCode) => {
      if (keyCode === BFKeyType.mode && !isTimeDone.current) {
        isSettingVoice.current = true;
        playPollyTTS({
          text: ' 지금부터, 음성 속도와 크기 조절을 시작합니다...',
          key: 'settingVoice',
        });
        return;
      }

      if (isSettingVoice.current && !isTimeDone.current) {
        if (keyCode === BFKeyType.center) {
          await stopCurrentAudio();
          if (narration) {
            playPollyTTS({
              text: '속도, 크기 조절 완료',
              key: 'settingVoice_done_',
            }).then(() => {
              setTimeout(() => {
                playPollyTTS({ text: narration.text, key: narration.key });
              }, 3000);
            });
          }
          isSettingVoice.current = false;
          return;
        }

        if (
          keyCode === BFKeyType.volumeUp ||
          keyCode === BFKeyType.volumeDown
        ) {
          settingVolume(keyCode, currentVolume.current);
          return;
        }

        if (
          keyCode === BFKeyType.arrowRight ||
          keyCode === BFKeyType.arrowLeft
        ) {
          settingSpeed(keyCode, currentSpeed.current);
          return;
        }

        if (keyCode === BFKeyType.arrowUp) {
          await stopCurrentAudio();
          replayLastAudio();
          return;
        }
      }
    },
    [narration, stopCurrentAudio],
  );

  const selectItem = React.useCallback(
    (keyCode: BFKeyCode) => {
      const validRefs = (focusableRefs?.current || []).filter(Boolean);
      if (!validRefs.length) return;

      if (keyCode === BFKeyType.arrowRight) {
        currentIndexRef.current =
          (currentIndexRef.current + 1) % validRefs.length;
      } else {
        currentIndexRef.current =
          currentIndexRef.current - 1 < 0
            ? validRefs.length - 1
            : currentIndexRef.current - 1;
      }

      validRefs[currentIndexRef.current]?.focus?.();
    },
    [focusableRefs],
  );

  const onPressItem = React.useCallback(() => {
    const validRefs = (focusableRefs?.current || []).filter(Boolean);
    if (!validRefs.length) return;
    validRefs[currentIndexRef.current]?.press?.();
  }, [focusableRefs]);

  React.useEffect(() => {
    if (!isFocused) return;

    setTimeout(() => {
      KeyEvent.onKeyDownListener((keyEvent) => {
        (async () => {
          if (!isFocused || !visible) return;
          setMilliseconds(0);
          setModalSeconds(0);

          if (!isSettingVoice.current && !isTimeDone.current) {
            switch (keyEvent.keyCode) {
              case BFKeyType.home:
                if (onHome) {
                  await stopCurrentAudio();
                  onHome();
                }
                break;
              case BFKeyType.info:
                if (onInfo) {
                  await stopCurrentAudio();
                  onInfo();
                } else {
                  await playPollyTTS({
                    text: '호출 선택',
                    key: 'on_info',
                  }).then(() => {
                    modal.open('bf/EmpCallModal', {}, {});
                  });
                }
                break;
              case BFKeyType.arrowUp:
                if (onArrowUp) {
                  await stopCurrentAudio();
                  onArrowUp();
                }
                break;
              case BFKeyType.volumeDown:
              case BFKeyType.volumeUp:
                await stopCurrentAudio();
                replayLastAudio();
                break;
              case BFKeyType.arrowRight:
              case BFKeyType.arrowLeft:
                if (onSelect) {
                  onSelect();
                } else {
                  selectItem(keyEvent.keyCode);
                }

                break;
              case BFKeyType.center:
                await stopCurrentAudio();
                onPressItem();
                break;
              case BFKeyType.back:
                if (onBack) {
                  await stopCurrentAudio();
                  onBack();
                }
                break;
            }
          }

          if (keyEvent.keyCode === BFKeyType.arrowDown && !isTimeDone.current) {
            await stopCurrentAudio();
            replayLastAudio();
          }

          await settingVoice(keyEvent.keyCode);

          if (isTimeDone.current) {
            if (
              onReset &&
              keyEvent.keyCode !== BFKeyType.jackIn &&
              keyEvent.keyCode !== BFKeyType.jackOut
            ) {
              await stopCurrentAudio();
              onReset();
            }
          }
        })();
      });
    }, 200);

    return () => {
      KeyEvent.removeKeyDownListener();
    };
  }, [isFocused, visible]);
};

 

키보드 훅도 해당 내용으로 작성했는데, 훅안에서 ref 값들을 배열로 받고, 키보드의 움직임에 따라 focus 를 호출하도록하였다.

 

단기간에, 큰 이슈없이 생각한대로 개발이 돼서 뿌듯하다

 

해당 코드는 회사 코드가 아니라 예시를 위해 제가 새로 작성한 코드입니다.